Try to use -location instead of -app -site. App exprected a virtual
path.

aspnet_regiis.exe /?

-pe section         Encrypt the configuration section. Optional
arguments:
                   [-prov provider] Use this provider to encrypt.
                   [-app virtual-path] Encrypt at this virtual path.
Virtual
                   path must begin with a forward slash. If it is
'/', then it
                   refers to the root of the site. If -app is not
specified,
                   the root web.config will be encrypted.
                   [-site site-name-or-ID] The site of the virtual
path
                   specified in -app. If not specified, the default
web site
                   will be used.
                   [-location sub-path] Location sub path.
                   [-pkm] Encrypt/decrypt the machine.config instead
of
                   web.config.

Did you try doing a aspnet_regiis -pef instead of -pe? You can then reference
the config file's physical path and avoid the hassle. I've always used that
anyways. To me it seems more precise and less blackboxish.