Since PowerShell 1.0, there has been a bug in Remove-ItemProperty that prevented you from deleting a registry keys’ default value. This bug still exists in PowerShell 3.0.
So if you must get rid of a default value, here is a (relatively) easy way that won’t require much extra work:
|# create a test key with default value:
New-Item -Path HKCU:\Testkey -Value 'somedefaultvalue'
# Remove-ItemProperty fails:
# delete default value on HKCU:\Testkey:
Getting rid of a registry key default value
Basically, you use Get-Item to get to the parent registry key. That’s a lot easier than using low level .NET because you can use PowerShell path syntax and won’t need the cryptic constants for Registry hives.
Next, you open the key in question with write permissions, then delete any value you want, including the default value (which happens to have no name).