The permissions for each assembly are granted based on its load location
(the directory where the .dll file is).  Assemblies in the global assembly
cache run with FullTrust, so they can call unmanaged code.  If the trusted
assembly is marked with AllowPartiallyTrustedCallersAttribute then untrusted
code can call the unmanaged code in whatever way the trusted assembly
allows.

None of this has anything to do with "safe" or "unsafe" though.  "unsafe"
only applies to pointer arithmetic, where you could overrun the bounds of an
array or cast to a different pointer type.  I believe it's actually an error
in the .NET specification that unsafe blocks are required for other pointer
operations, but clearly someone at Microsoft really doesn't like pointers.