Additional source-header notices, retained verbatim below their source locations.

Source: build.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

// The rustc-cfg emitted by the build script are *not* public API.


Source: src/cfgs.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/gen/build.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT
// This file is @generated by target_spec.sh.
// It is not intended for manual editing.

// Note: This is the list as of nightly-2022-02-10. We don't refer to this in
// nightly-2022-02-11+ because feature(cfg_target_has_atomic) stabilized.

Source: src/gen/utils.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT
// This file is @generated by target_spec.sh.
// It is not intended for manual editing.


Source: src/imp/arm_linux.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic operations implementation on pre-v6 ARM Linux/Android.

This module optimizes cases where LLVM code generation is not optimal,
and provides workaround for core::sync::atomic bug (https://github.com/rust-lang/compiler-builtins/pull/1234).

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#arm

Refs:
- https://github.com/torvalds/linux/blob/v6.19/Documentation/arch/arm/kernel_user_helpers.rst
- https://github.com/rust-lang/compiler-builtins/blob/compiler_builtins-v0.1.124/src/arm_linux.rs
- https://github.com/taiki-e/atomic-maybe-uninit

Note: __kuser_memory_barrier is always SeqCst.
https://github.com/taiki-e/atomic-maybe-uninit/blob/v0.3.21/src/arch/arm.rs#L215

Note: On Miri and ThreadSanitizer which do not go through compiler-builtins, we don't use
this module and use normal core::sync::atomic instead.
*/


Source: src/imp/atomic128/aarch64.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementation on AArch64.

This architecture provides the following 128-bit atomic instructions:

- LDXP/STXP: LL/SC (Armv8.0 baseline)
- CASP: CAS (added as Armv8.1 FEAT_LSE (optional from Armv8.0, mandatory from Armv8.1))
- LDP/STP: load/store (if Armv8.4 FEAT_LSE2 (optional from Armv8.2, mandatory from Armv8.4) is available)
- LDIAPP/STILP: acquire-load/release-store (added as Armv8.9 FEAT_LRCPC3 (optional from Armv8.2) (if FEAT_LSE2 is also available))
- LDCLRP/LDSETP/SWPP: fetch-and-{clear,or},swap (added as Armv9.4 FEAT_LSE128 (optional from Armv9.3))

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#aarch64

This module supports all of these instructions and attempts to select the best
one based on compile-time and run-time information about available CPU features
and platforms. For example:

- If outline-atomics is not enabled and FEAT_LSE is not available at
  compile-time, we use LDXP/STXP loop.
- If outline-atomics is enabled and FEAT_LSE is not available at
  compile-time, we use CASP for CAS if FEAT_LSE is available
  at run-time, otherwise, use LDXP/STXP loop.
- If FEAT_LSE is available at compile-time, we use CASP for load/store/CAS/RMW.
  However, when portable_atomic_ll_sc_rmw cfg is set, use LDXP/STXP loop instead of CASP
  loop for RMW (by default, it is set on Apple hardware where CASP is slow;
  see build script for details).
- If outline-atomics is enabled and FEAT_LSE2 is not available at compile-time,
  we use LDP/STP (and also LDIAPP/STILP/SWPP if FEAT_LRCPC3/FEAT_LSE128 is
  available) for load/store if FEAT_LSE2 is available at run-time, otherwise,
  use LDXP/STXP or CASP depending on whether FEAT_LSE is available.
- If FEAT_LSE2 is available at compile-time, we use LDP/STP for load/store.
- If FEAT_LSE128 is available at compile-time, we use LDCLRP/LDSETP/SWPP for fetch_and/fetch_or/swap/{release,seqcst}-store.
- If FEAT_LSE2 and FEAT_LRCPC3 are available at compile-time, we use LDIAPP/STILP for acquire-load/release-store.

See each "Instruction selection flow for ..." comment in this file for the exact
instruction selection per operation.

Note: FEAT_LSE2 doesn't imply FEAT_LSE. FEAT_LSE128 implies FEAT_LSE but not FEAT_LSE2.

Note that we do not separate LL and SC into separate functions, but handle
them within a single asm block. This is because it is theoretically possible
for the compiler to insert operations that might clear the reservation between
LL and SC. Considering the type of operations we are providing and the fact
that [progress64](https://github.com/ARM-software/progress64) uses such code,
this is probably not a problem for AArch64, but it seems that AArch64 doesn't
guarantee it and hexagon is the only architecture with hardware guarantees
that such code works. See also:

- https://yarchive.net/comp/linux/cmpxchg_ll_sc_portability.html
- https://lists.llvm.org/pipermail/llvm-dev/2016-May/099490.html
- https://lists.llvm.org/pipermail/llvm-dev/2018-June/123993.html

Also, even when using a CAS loop to implement atomic RMW, include the loop itself
in the asm block because it is more efficient for some codegen backends.
https://github.com/rust-lang/compiler-builtins/issues/339#issuecomment-1191260474

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use intrinsics.rs instead.

Refs:
- Arm A-profile A64 Instruction Set Architecture
  https://developer.arm.com/documentation/ddi0602/2025-06
- C/C++ Atomics Application Binary Interface Standard for the Arm® 64-bit Architecture
  https://github.com/ARM-software/abi-aa/blob/2025Q4/atomicsabi64/atomicsabi64.rst
- Arm Compiler armasm User Guide
  https://developer.arm.com/documentation/dui0801/latest
- Arm Architecture Reference Manual for A-profile architecture
  https://developer.arm.com/documentation/ddi0487/latest (PDF)
- Arm Architecture Reference Manual Supplement - Armv8, for Armv8-R AArch64 architecture profile
  https://developer.arm.com/documentation/ddi0600/latest (PDF)
- atomic-maybe-uninit https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/atomic128/intrinsics.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementation without inline assembly.

Adapted from https://github.com/rust-lang/rust/blob/1.84.0/library/core/src/sync/atomic.rs.

Note: This module is currently only enabled on Miri and ThreadSanitizer which
do not support inline assembly.

This uses `core::arch::x86_64::cmpxchg16b` on x86_64 and
`core::intrinsics::atomic_*` on aarch64, powerpc64, and s390x.

See README.md of this directory for performance comparison with the
implementation with inline assembly.

Note:
- This currently needs Rust 1.70 on x86_64, otherwise nightly compilers.
- On powerpc64, this requires LLVM 15+ and quadword-atomics target feature:
  https://github.com/llvm/llvm-project/commit/549e118e93c666914a1045fde38a2cac33e1e445
- On s390x, old LLVM (pre-18) generates libcalls for operations other than load/store/cmpxchg:
  https://github.com/llvm/llvm-project/commit/c568927f3e2e7d9804ea74ecbf11c16c014ddcbc
- On aarch64 big-endian, LLVM (as of 17) generates broken code. (wrong result in stress test)
  (on cfg(miri)/cfg(sanitize) it may be fine though)
- On powerpc64, LLVM (as of 17) doesn't support 128-bit atomic min/max:
  https://github.com/llvm/llvm-project/issues/68390
- On powerpc64le, LLVM (as of 17) generates broken code. (wrong result from fetch_add)
- On riscv64, LLVM does not automatically use 128-bit atomic instructions even if zacas feature is
  enabled, because doing it changes the ABI. (If the ability to do that is provided by LLVM in the
  future, it should probably be controlled by another ABI feature similar to forced-atomics.)
*/


Source: src/imp/atomic128/macros.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/imp/atomic128/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementations on 64-bit architectures

See README.md for details.
*/

// AArch64

Source: src/imp/atomic128/powerpc64.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementation on PowerPC64.

This architecture provides the following 128-bit atomic instructions:

- lq/stq: load/store (ISA 2.07 or later, included in the Linux Compliancy subset and AIX Compliancy subset)
- lqarx/stqcx.: LL/SC (ISA 2.07 or later, included in the Linux Compliancy subset and AIX Compliancy subset)
- plq/pstq: load/store (ISA 3.1 or later, included in the Linux Compliancy subset and AIX Compliancy subset)

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#powerpc

Note that we do not separate LL and SC into separate functions, but handle
them within a single asm block. This is because it is theoretically possible
for the compiler to insert operations that might clear the reservation between
LL and SC. See aarch64.rs for details.

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use intrinsics.rs instead.

Refs:
- Power ISA
  https://openpowerfoundation.org/specifications/isa
- AIX Assembler language reference
  https://www.ibm.com/docs/en/aix/7.3?topic=aix-assembler-language-reference
- atomic-maybe-uninit
  https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/atomic128/riscv64.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementation on riscv64.

This architecture provides the following 128-bit atomic instructions:

- amocas.q: CAS (Zacas extension)

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#risc-v

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use fallback implementation instead.

Refs:
- RISC-V Instruction Set Manual
  "Zacas" Extension for Atomic Compare-and-Swap (CAS) Instructions
  https://docs.riscv.org/reference/isa/v20260120/unpriv/zacas.html
- RISC-V Atomics ABI Specification
  https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20250812-301374e92976e298e676e7129a6212926b2299ce/riscv-atomic.adoc

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/atomic128/s390x.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementation on s390x.

This architecture provides the following 128-bit atomic instructions:

- LPQ/STPQ: load/store (arch1 or later, i.e., baseline)
- CDSG: CAS (arch1 or later, i.e., baseline)

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#s390x

LLVM's minimal supported architecture level is arch8 (z10):
https://github.com/llvm/llvm-project/blob/llvmorg-22.1.0-rc1/llvm/lib/Target/SystemZ/SystemZProcessors.td#L16-L17
This does not appear to have changed since the current s390x backend was added in LLVM 3.3:
https://github.com/llvm/llvm-project/commit/5f613dfd1f7edb0ae95d521b7107b582d9df5103#diff-cbaef692b3958312e80fd5507a7e2aff071f1acb086f10e8a96bc06a7bb289db

We could use asm byte_wise_atomic_load + asm atomic_compare_exchange pattern for RMW here, but using an
inline assembly allows omitting the comparison of results and the storing/comparing of condition flags.

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use intrinsics.rs instead.

Refs:
- z/Architecture Principles of Operation, Fifteenth Edition (SA22-7832-14)
  https://www.ibm.com/docs/en/module_1678991624569/pdf/SA22-7832-14.pdf
- atomic-maybe-uninit
  https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/atomic128/x86_64.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
128-bit atomic implementation on x86_64.

This architecture provides the following 128-bit atomic instructions:

- CMPXCHG16B: CAS (CMPXCHG16B)
- VMOVDQA: load/store (AVX)

We could use asm byte_wise_atomic_load + asm atomic_compare_exchange pattern for RMW here, but using an
inline assembly allows omitting the comparison of results and the storing/comparing of flags.

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use intrinsics.rs instead.

Refs:
- x86 and amd64 instruction reference https://www.felixcloutier.com/x86
- atomic-maybe-uninit https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/

// TODO: use core::arch::x86_64::cmpxchg16b where available and efficient than asm


Source: src/imp/atomic64/arm_linux.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
64-bit atomic implementation using kuser_cmpxchg64 on pre-v6 Arm Linux/Android.

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#arm

Refs:
- https://github.com/torvalds/linux/blob/v6.19/Documentation/arch/arm/kernel_user_helpers.rst
- https://github.com/rust-lang/compiler-builtins/blob/compiler_builtins-v0.1.124/src/arm_linux.rs
- https://github.com/taiki-e/atomic-maybe-uninit

Note: __kuser_cmpxchg64 is always SeqCst.
https://github.com/taiki-e/atomic-maybe-uninit/blob/v0.3.21/src/arch/arm.rs#L230

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use fallback implementation instead.
*/

// TODO: Since Rust 1.64, the Linux kernel requirement for Rust when using std is 3.2+, so it should
// be possible to omit the dynamic kernel version check if the std feature is enabled on Rust 1.64+.
// https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements


Source: src/imp/atomic64/macros.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/imp/atomic64/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
64-bit atomic implementations on 32-bit architectures

See README.md for details.
*/

// pre-v6 Arm Linux
// Miri and Sanitizer do not support inline assembly.

Source: src/imp/atomic64/riscv32.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
64-bit atomic implementation on riscv32.

This architecture provides the following 64-bit atomic instructions:

- amocas.d: CAS (Zacas extension)

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#risc-v

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use fallback implementation instead.

Refs:
- RISC-V Instruction Set Manual
  "Zacas" Extension for Atomic Compare-and-Swap (CAS) Instructions
  https://docs.riscv.org/reference/isa/v20260120/unpriv/zacas.html
- RISC-V Atomics ABI Specification
  https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20250812-301374e92976e298e676e7129a6212926b2299ce/riscv-atomic.adoc

See tests/asm-test/asm/portable-atomic for generated assembly.
*/

// TODO: merge duplicated code with atomic128/riscv64.rs


Source: src/imp/avr.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic load/store implementation on AVR.

Operations not supported here are provided by disabling interrupts.
See also src/imp/interrupt/avr.rs.

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#avr

Note: Ordering is always SeqCst.

Refs:
- AVR® Instruction Set Manual, Rev. DS40002198B
  https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf
- atomic-maybe-uninit
  https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/core_atomic.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Wrap the standard library's atomic types in newtype.

This is not a reexport, because we want to backport changes like
https://github.com/rust-lang/rust/pull/98383 to old compilers.
*/


Source: src/imp/detect/aarch64_aa64reg.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on AArch64 Linux/Android/FreeBSD/NetBSD/OpenBSD by parsing system registers.

As of Rust 1.94, is_aarch64_feature_detected doesn't support run-time detection on NetBSD.
https://github.com/rust-lang/rust/blob/1.94.0/library/std_detect/src/detect/mod.rs
Run-time detection on OpenBSD by is_aarch64_feature_detected is supported on Rust 1.70+.
https://github.com/rust-lang/stdarch/pull/1374

Refs:
- https://developer.arm.com/documentation/ddi0601/2025-06/AArch64-Registers
- https://github.com/torvalds/linux/blob/v6.19/Documentation/arch/arm64/cpu-feature-registers.rst
- https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/os/aarch64.rs

Supported platforms:
- Linux 4.11+ (emulate mrs instruction)
  https://github.com/torvalds/linux/commit/77c97b4ee21290f5f083173d957843b615abbff2
- FreeBSD 12.0+ (emulate mrs instruction)
  https://github.com/freebsd/freebsd-src/commit/398810619cb32abf349f8de23f29510b2ee0839b
- NetBSD 9.0+ (through sysctl/sysctlbyname)
  https://github.com/NetBSD/src/commit/0e9d25528729f7fea53e78275d1bc5039dfe8ffb
  sysctl/sysctlbyname returns an unsupported error if operation is not supported,
  so we can safely use this on older versions.
- OpenBSD 7.1+ (through sysctl)
  https://github.com/openbsd/src/commit/d335af936b9d7dd9cf655cae1ce19560c45de6c8
  sysctl returns an unsupported error if operation is not supported,
  so we can safely use this on older versions.

For now, this module is only used on NetBSD/OpenBSD.

On Linux/Android/FreeBSD, we use auxv.rs and this module is test-only because:
- On Linux/Android, this approach requires a higher kernel version than Rust supports,
  and also does not work with qemu-user (as of QEMU 7.2) and Valgrind (as of Valgrind 3.24).
  (Looking into HWCAP_CPUID in auxvec, it appears that Valgrind is setting it
  to false correctly, but qemu-user is setting it to true.)
  - qemu-user issue seem to be fixed as of QEMU 9.2.
- On FreeBSD, this approach does not work on FreeBSD 12 on QEMU (confirmed on
  FreeBSD 12.{2,3,4}), and we got SIGILL (worked on FreeBSD 13 and 14).
*/


Source: src/imp/detect/aarch64_apple.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on AArch64 Apple targets by using sysctlbyname.

On macOS, this module is currently only enabled on tests because there are no
instructions that were not available on the M1 but are now available on the
latest Apple hardware and this library currently wants to use:

```console
$ LC_ALL=C comm -23 <(rustc --print cfg --target aarch64-apple-darwin -C target-cpu=apple-m5 | grep -F target_feature) <(rustc --print cfg --target aarch64-apple-darwin | grep -F target_feature)
target_feature="bf16"
target_feature="bti"
target_feature="cssc"
target_feature="ecv"
target_feature="hbc"
target_feature="i8mm"
target_feature="mte"
target_feature="sme"
target_feature="sme-b16b16"
target_feature="sme-f16f16"
target_feature="sme-f64f64"
target_feature="sme-i16i64"
target_feature="sme2"
target_feature="sme2p1"
target_feature="sve-b16b16"
target_feature="v8.5a"
target_feature="v8.6a"
target_feature="v8.7a"
target_feature="wfxt"
```

Non-macOS targets doesn't always supports FEAT_LSE2, so we use this module on them.
As used in the example in https://developer.apple.com/documentation/xcode/writing-arm64-code-for-apple-platforms#Enable-DIT-for-constant-time-cryptographic-operations,
sysctlbyname is supported on all Apple platforms.

Refs: https://developer.apple.com/documentation/kernel/1387446-sysctlbyname/determining_instruction_set_characteristics
*/


Source: src/imp/detect/aarch64_fuchsia.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on AArch64 Fuchsia by using zx_system_get_features.

As of Rust 1.94, is_aarch64_feature_detected doesn't support run-time detection on Fuchsia.
https://github.com/rust-lang/rust/blob/1.94.0/library/std_detect/src/detect/mod.rs

Refs:
- https://fuchsia.dev/reference/syscalls/system_get_features
- https://github.com/llvm/llvm-project/commit/4e731abc55681751b5d736b613f7720e50eb1ad4
*/


Source: src/imp/detect/aarch64_illumos.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on AArch64 illumos by using getisax.

As of Rust 1.94, is_aarch64_feature_detected doesn't support run-time detection on illumos.
https://github.com/rust-lang/rust/blob/1.94.0/library/std_detect/src/detect/mod.rs
*/


Source: src/imp/detect/aarch64_windows.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on AArch64 Windows by using IsProcessorFeaturePresent.

Run-time detection of FEAT_LSE on Windows by is_aarch64_feature_detected is supported on Rust 1.70+.
https://github.com/rust-lang/stdarch/pull/1373

Refs: https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-isprocessorfeaturepresent
*/


Source: src/imp/detect/auxv.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on AArch64/Arm/PowerPC64 Linux/Android/FreeBSD/OpenBSD by parsing ELF auxiliary vectors.

Supported platforms:
- Linux 6.4+ (through prctl)
  https://github.com/torvalds/linux/commit/ddc65971bb677aa9f6a4c21f76d3133e106f88eb
  prctl returns an unsupported error if operation is not supported,
  so we can safely use this on older versions.
- glibc 2.16+ (through getauxval)
  https://sourceware.org/git/?p=glibc.git;a=blob;f=NEWS;hb=glibc-2.16#l83
  Always available on:
  - aarch64 (glibc 2.17+ https://sourceware.org/git/?p=glibc.git;a=blob;f=NEWS;hb=glibc-2.17#l36)
  - powerpc64 (le) (glibc 2.19+ or RHEL/CentOS's patched glibc 2.17+ https://sourceware.org/git/?p=glibc.git;a=blob;f=NEWS;hb=glibc-2.19#l108)
  Not always available on:
  - arm (glibc 2.1+ https://sourceware.org/git/?p=glibc.git;a=blob;f=NEWS;hb=glibc-2.1#l97)
  - powerpc64 (be) (glibc 2.3+ https://sourceware.org/git/?p=glibc.git;a=blob;f=NEWS;hb=glibc-2.3#l56)
  Since Rust 1.64, std requires glibc 2.17+ https://blog.rust-lang.org/2022/08/01/Increasing-glibc-kernel-requirements
- musl 1.1.0+ (through getauxval)
  https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v1.1.0#n1197
  Always available on:
  - aarch64 (musl 1.1.7+ https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v1.1.7#n1422)
  - powerpc64 (musl 1.1.15+ https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v1.1.15#n1702)
  Not always available on:
  - arm (musl 0.8.3+ https://git.musl-libc.org/cgit/musl/tree/WHATSNEW?h=v0.8.3#n354)
  At least since Rust 1.15, std requires musl 1.1.14+ https://github.com/rust-lang/rust/blob/1.15.0/src/ci/docker/x86_64-musl/build-musl.sh#L15
  Since Rust 1.18, std requires musl 1.1.16+ https://github.com/rust-lang/rust/pull/41089
  Since Rust 1.23, std requires musl 1.1.17+ https://github.com/rust-lang/rust/pull/45393
  Since Rust 1.25, std requires musl 1.1.18+ https://github.com/rust-lang/rust/pull/47283
  Since Rust 1.29, std requires musl 1.1.19+ https://github.com/rust-lang/rust/pull/52087
  Since Rust 1.31, std requires musl 1.1.20+ https://github.com/rust-lang/rust/pull/54430
  Since Rust 1.37, std requires musl 1.1.22+ https://github.com/rust-lang/rust/pull/61252
  Since Rust 1.46, std requires musl 1.1.24+ https://github.com/rust-lang/rust/pull/73089
  Since Rust 1.71, std requires musl 1.2.3+ https://blog.rust-lang.org/2023/05/09/Updating-musl-targets
  Since Rust 1.93, std requires musl 1.2.5+ https://blog.rust-lang.org/2025/12/05/Updating-musl-1.2.5/
  OpenHarmony uses a fork of musl 1.2 https://gitee.com/openharmony/docs/blob/master/en/application-dev/reference/native-lib/musl.md
- uClibc-ng 1.0.43+ (through getauxval)
  https://github.com/wbx-github/uclibc-ng/commit/d869bb1600942c01a77539128f9ba5b5b55ad647
  Not always available on:
  - aarch64 (uClibc-ng 1.0.22+ https://github.com/wbx-github/uclibc-ng/commit/dba942c80dc2cfa5768a856fff98e22a755fdd27)
  - arm (uClibc-ng 1.0.0+ https://github.com/wbx-github/uclibc-ng/tree/v1.0.0/libc/sysdeps/linux)
  (powerpc64 is not supported https://github.com/wbx-github/uclibc-ng/commit/d4d4f37fda7fa57e57132ff2f0d735ce7cc2178e)
- Picolibc 1.4.6+ (through getauxval)
  https://github.com/picolibc/picolibc/commit/19bfe51d62ad7e32533c7f664b5bca8e26286e31
  The implementation always return 0 (as of 1.8.12): https://github.com/picolibc/picolibc/blob/1.8.12/libc/misc/getauxval.c
- Android 4.3+ (API level 18+) (through getauxval)
  https://android.googlesource.com/platform/bionic.git/+/2c5153b043b44e9935a334ae9b2d5a4bc5258b40
  https://android.googlesource.com/platform/bionic.git/+/655e430b28d7404f763e7ebefe84fba5a387666d
  Always available on:
  - 64-bit architectures (Android 5.0+ (API level 21+) https://android-developers.googleblog.com/2014/10/whats-new-in-android-50-lollipop.html)
  Not always available on:
  - arm
  Since Rust 1.68, std requires API level 19+ https://blog.rust-lang.org/2023/01/09/android-ndk-update-r25
  Since Rust 1.82, std requires API level 21+ https://github.com/rust-lang/rust/pull/120593
- FreeBSD 12.0+ and 11.4+ (through elf_aux_info)
  https://github.com/freebsd/freebsd-src/commit/0b08ae2120cdd08c20a2b806e2fcef4d0a36c470
  https://github.com/freebsd/freebsd-src/blob/release/11.4.0/sys/sys/auxv.h
  Always available on:
  - arm (v7) (FreeBSD 12.0+ https://www.freebsd.org/releases/12.0R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
  - powerpc64 (le) (FreeBSD 12.4+ https://www.freebsd.org/releases/12.4R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
  Not always available on:
  - aarch64 (FreeBSD 11.0+ https://www.freebsd.org/releases/11.0R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
  - arm (v6) (FreeBSD 10.1+ https://www.freebsd.org/releases/10.1R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
  - powerpc64 (be) (FreeBSD 9.0+ https://www.freebsd.org/releases/9.0R/announce, https://man.freebsd.org/cgi/man.cgi?arch)
  Since Rust 1.75, std requires FreeBSD 12+ https://github.com/rust-lang/rust/pull/114521
  Since Rust 1.97, std requires FreeBSD 14+ https://github.com/rust-lang/rust/pull/156607
- OpenBSD 7.6+ (through elf_aux_info)
  https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d
  Not always available on:
  - aarch64 (OpenBSD 6.1+ https://www.openbsd.org/61.html)
  - arm (OpenBSD 6.0+ https://www.openbsd.org/60.html)
  - powerpc64 (OpenBSD 6.8+ https://www.openbsd.org/68.html)

On L4Re which uses uClibc-ng, a dummy getauxval was added first in 2020 that always returns 0 (https://github.com/kernkonzept/l4re-core/commit/e88fa67198074d3e6b4983c5c8af1538e2089ff3),
then implemented in 2024 (https://github.com/kernkonzept/l4re-core/commit/3ee2a50dd1b3bc22955e593004990887a0a5b4a3).
However, getauxval(AT_HWCAP*) always returns 0 (as of 2025-12-25). (see tests/l4re test)
On Redox, getauxval is available since 0.5.0 (https://github.com/redox-os/relibc/commit/f9f752d74c4f1f56a89c0fcdd5cab63d2380fe09),
but the implementation always return 0 (as of 2026-08-08): https://github.com/redox-os/relibc/blob/49aeb9470a885030ed43a70d0b708f3caec19478/src/header/sys_auxv/mod.rs

On platforms that we can assume that getauxval/elf_aux_info is always available, we directly call
them on except for musl with static linking. (At this time, we also retain compatibility with
versions that reached EoL or no longer supported by `std`, with the exception of AArch64 FreeBSD described below.)

On musl with static linking, it seems that getauxval is not always available, independent of version
requirements: https://github.com/rust-lang/rust/issues/89626
(That problem may have been fixed in https://github.com/rust-lang/rust/commit/9a04ae4997493e9260352064163285cddc43de3c,
but even in the version containing that patch, [there is report](https://github.com/rust-lang/rust/issues/89626#issuecomment-1242636038)
of the same error.)
This seems to be due to the fact that compiler-builtins is built before libc (which is a dependency
of std) links musl. And the std and its dependent can use getauxval without this problem at least
since the rust-lang/rust patch mentioned above:
https://github.com/rust-lang/rust/blob/1.85.0/library/std/src/sys/pal/unix/stack_overflow.rs#L268
(According to https://github.com/rust-lang/rust/issues/89626#issuecomment-2420469392, this problem
may have been fixed in https://github.com/rust-lang/rust/commit/9ed0d11efbec18a1fa4155576a3bcb685676d23c.)
See also https://github.com/rust-lang/stdarch/pull/1746.
So as for musl with static linking, we assume that getauxval is always available also when `std` feature enabled.

On platforms that we cannot assume that getauxval/elf_aux_info is always available, so we use dlsym
instead of directly calling getauxval/elf_aux_info. (You can force getauxval/elf_aux_info to be
called directly instead of using dlsym by `--cfg portable_atomic_outline_atomics`).

Also, note that dlsym usually not working with static linking.

# Linux/Android

As of Rust 1.69, is_aarch64_feature_detected always uses dlsym by default
on AArch64 Linux/Android, but on some platforms, we can safely assume
getauxval is linked to the binary (see the above).

See also https://github.com/rust-lang/stdarch/pull/1375

See tests::test_alternative and aarch64_aa64reg.rs for (test-only) alternative implementations.

# FreeBSD

As of Rust 1.94, is_aarch64_feature_detected always uses mrs on
AArch64 FreeBSD. However, they do not work on FreeBSD 12 on QEMU (confirmed
on FreeBSD 12.{2,3,4}), and we got SIGILL (worked on FreeBSD 13 and 14).

So use elf_aux_info instead of mrs like compiler-rt does.
https://reviews.llvm.org/D109330

elf_aux_info is available on FreeBSD 12.0+ and 11.4+:
https://github.com/freebsd/freebsd-src/commit/0b08ae2120cdd08c20a2b806e2fcef4d0a36c470
https://github.com/freebsd/freebsd-src/blob/release/11.4.0/sys/sys/auxv.h
On FreeBSD, [AArch64 support is available on FreeBSD 11.0+](https://www.freebsd.org/releases/11.0R/announce),
but FreeBSD 11 (11.4) was EoL on 2021-09-30, and FreeBSD 11.3 was EoL on 2020-09-30:
https://www.freebsd.org/security/unsupported
See also https://github.com/rust-lang/stdarch/pull/611#issuecomment-445464613

See tests::test_alternative and aarch64_aa64reg.rs for (test-only) alternative implementations.

# OpenBSD

elf_aux_info is available on OpenBSD 7.6+:
https://github.com/openbsd/src/commit/ef873df06dac50249b2dd380dc6100eee3b0d23d

On AArch64, there is an alternative that available on older version,
so we use it (see aarch64_aa64reg.rs).

*/


Source: src/imp/detect/common.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/imp/detect/powerpc64_aix.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on PowerPC64 AIX by using getsystemcfg.

Refs:
- https://github.com/golang/go/blob/go1.26.0/src/internal/cpu/cpu_ppc64x_aix.go

As of Rust 1.94, is_powerpc_feature_detected doesn't support run-time detection on AIX.
https://github.com/rust-lang/rust/blob/1.94.0/library/std_detect/src/detect/mod.rs

Run-time detection on PowerPC64 AIX is currently disabled by default as experimental
because we cannot run tests on the VM or real machine.
*/


Source: src/imp/detect/riscv_linux.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on RISC-V Linux/Android by using riscv_hwprobe.

On RISC-V, detection using auxv only supports single-letter extensions.
So, we use riscv_hwprobe that supports multi-letter extensions.

Refs: https://github.com/torvalds/linux/blob/v6.19/Documentation/arch/riscv/hwprobe.rst
*/


Source: src/imp/detect/x86_64.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Run-time CPU feature detection on x86_64 by using CPUID.

Adapted from https://github.com/rust-lang/rust/blob/1.92.0/library/std_detect/src/detect/os/x86.rs.
*/


Source: src/imp/fallback/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Fallback implementation using global locks.

This implementation uses seqlock for global locks.

This is basically based on global locks in crossbeam-utils's `AtomicCell`,
but seqlock is implemented in a way that does not depend on UB
(see comments in optimistic_read method in atomic! macro for details).

Note that we cannot use a lock per atomic type, since the in-memory representation of the atomic
type and the value type must be the same.
*/


Source: src/imp/fallback/outline_atomics.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Helper for outline-atomics.

On architectures where DW atomics are not supported on older CPUs, we use
fallback implementation when DW atomic instructions are not supported and
outline-atomics is enabled.

This module provides helpers to implement them.
*/


Source: src/imp/fallback/seq_lock.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/seq_lock.rs.


Source: src/imp/fallback/seq_lock_common.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/seq_lock.rs
// and https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/seq_lock_wide.rs.


Source: src/imp/fallback/seq_lock_wide.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.21/crossbeam-utils/src/atomic/seq_lock_wide.rs.


Source: src/imp/fallback/utils.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/imp/float/aarch64.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic float implementation based on AArch64 with FEAT_LSFE.

This module provides atomic float implementations using FEAT_LSFE instructions.

Generated asm:
- aarch64 (+lsfe) https://godbolt.org/z/7vaxeofv1
*/


Source: src/imp/float/int.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic float implementation based on atomic integer.

This module provides atomic float implementations using atomic integer.

Note that most of `fetch_*` operations of atomic floats are implemented using
CAS loops, which can be slower than equivalent operations of atomic integers.

AArch64 with FEAT_LSFE and GPU targets have atomic instructions for float.
See aarch64.rs for AArch64 with FEAT_LSFE.
GPU targets will also use architecture-specific implementations instead of this implementation in the
future: https://github.com/taiki-e/portable-atomic/issues/34 / https://github.com/taiki-e/portable-atomic/pull/45
*/

// TODO: fetch_{minimum,maximum}* https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2024/p3008r2.html / https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2024/p0493r5.pdf


Source: src/imp/float/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic float implementations
*/


Source: src/imp/interrupt/armv4t.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Arm A-Profile Architectures, Arm R-Profile Architectures, Legacy Arm Architectures

Refs: https://developer.arm.com/documentation/ddi0406/cb/System-Level-Architecture/The-System-Level-Programmers--Model/ARM-processor-modes-and-ARM-core-registers/Program-Status-Registers--PSRs-

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/interrupt/armv6m.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Arm M-Profile Architectures

Adapted from https://github.com/rust-embedded/cortex-m.

Refs: https://developer.arm.com/documentation/ddi0419/c/System-Level-Architecture/System-Level-Programmers--Model/Registers/The-special-purpose-mask-register--PRIMASK

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/interrupt/avr.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Adapted from https://github.com/Rahix/avr-device.

See also src/imp/avr.rs.

Refs:
- AVR® Instruction Set Manual, Rev. DS40002198B
  https://ww1.microchip.com/downloads/en/DeviceDoc/AVR-InstructionSet-Manual-DS40002198.pdf

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/interrupt/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Fallback implementation based on disabling interrupts or critical-section

- mod.rs contains critical section based fallback implementations.
- Each architecture modules contain implementations of disabling interrupts.

See README.md of this directory for details.
*/


Source: src/imp/interrupt/msp430.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Adapted from https://github.com/rust-embedded/msp430.

See also src/imp/msp430.rs.

Refs:
- MSP430x5xx and MSP430x6xx Family User's Guide, Rev. Q
  https://www.ti.com/lit/ug/slau208q/slau208q.pdf

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/interrupt/riscv.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Refs:
- RISC-V Instruction Set Manual
  Machine Status (mstatus and mstatush) Registers
  https://docs.riscv.org/reference/isa/v20260120/priv/machine.html#3-1-1-6-machine-status-mstatus-and-mstatush-registers
  Supervisor Status (sstatus) Register
  https://docs.riscv.org/reference/isa/v20260120/priv/supervisor.html#sstatus

See also src/imp/riscv.rs.

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/interrupt/xtensa.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Refs:
- Xtensa Instruction Set Architecture (ISA) Summary for all Xtensa LX Processors
  https://www.cadence.com/content/dam/cadence-www/global/en_US/documents/tools/silicon-solutions/compute-ip/isa-summary.pdf
- Linux kernel's Xtensa atomic implementation
  https://github.com/torvalds/linux/blob/v6.19/arch/xtensa/include/asm/atomic.h

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

// -----------------------------------------------------------------------------
// Lock-free implementations


Source: src/imp/msp430.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic implementation on MSP430.

Adapted from https://github.com/pftbest/msp430-atomic.

Operations not supported here are provided by disabling interrupts.
See also src/imp/interrupt/msp430.rs.

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#msp430

Note: Ordering is always SeqCst.

Refs:
- MSP430x5xx and MSP430x6xx Family User's Guide, Rev. Q
  https://www.ti.com/lit/ug/slau208q/slau208q.pdf
- atomic-maybe-uninit
  https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/riscv.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic load/store implementation on RISC-V.

This is for RISC-V targets without A extension. (pre-1.76 rustc doesn't provide atomics
at all on such targets. https://github.com/rust-lang/rust/pull/114499)

Also, optionally provides RMW implementation when Zaamo extension or force-amo feature is enabled.

If A extension is not available but Zalrsc or Zacas extension is available, LLVM can implement all
atomic operations using it (https://github.com/llvm/llvm-project/commit/70f70390bb83a383fcb88ea843adbc4edf6ee34b),
so this module contains no code for these extensions.

See "Atomic operation overview by architecture" in atomic-maybe-uninit for a more comprehensive and
detailed description of the atomic and synchronize instructions in this architecture:
https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md#risc-v

Refs:
- RISC-V Instruction Set Manual
  "Zaamo" Extension for Atomic Memory Operations
  https://docs.riscv.org/reference/isa/v20260120/unpriv/a-st-ext.html#sec:amo
  "Zabha" Extension for Byte and Halfword Atomic Memory Operations
  https://docs.riscv.org/reference/isa/v20260120/unpriv/zabha.html
- RISC-V Atomics ABI Specification
  https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/draft-20250812-301374e92976e298e676e7129a6212926b2299ce/riscv-atomic.adoc
- atomic-maybe-uninit
  https://github.com/taiki-e/atomic-maybe-uninit

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/imp/x86.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*
Atomic operations implementation on x86/x86_64.

This module provides atomic operations not supported by LLVM or optimizes
cases where LLVM code generation is not optimal.

Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
this module and use CAS loop instead.

Refs:
- x86 and amd64 instruction reference https://www.felixcloutier.com/x86

See tests/asm-test/asm/portable-atomic for generated assembly.
*/


Source: src/lib.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT

/*!
<!-- Note: Document from sync-markdown-to-rustdoc:start through sync-markdown-to-rustdoc:end
     is synchronized from README.md. Any changes to that range are not preserved. -->
<!-- tidy:sync-markdown-to-rustdoc:start -->

Portable atomic types including support for 128-bit atomics, atomic float, etc.

- Provide all atomic integer types (`Atomic{I,U}{8,16,32,64}`) for all targets that can use atomic CAS. (i.e., all targets that can use `std`, and most no-std targets)
- Provide `AtomicI128` and `AtomicU128`.
- Provide `AtomicF32` and `AtomicF64`. ([optional, requires the `float` feature](#optional-features-float))
- Provide `AtomicF16` and `AtomicF128` for [unstable `f16` and `f128`](https://github.com/rust-lang/rust/issues/116909). ([optional, requires the `float` feature and unstable cfgs](#optional-features-float))
- Provide atomic load/store for targets where atomic is not available at all in the standard library. (RISC-V without A-extension, MSP430, AVR)
- Provide atomic CAS for targets where atomic CAS is not available in the standard library. (thumbv6m, pre-v6 Arm, RISC-V without A-extension, MSP430, AVR, Xtensa, etc.) (always enabled for MSP430 and AVR, [optional](#optional-features-critical-section) otherwise)
- Make features that require newer compilers, such as [`fetch_{max,min}`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.fetch_max), [`fetch_update`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.fetch_update), [`as_ptr`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.as_ptr), [`from_ptr`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicUsize.html#method.from_ptr), [`AtomicBool::fetch_not`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicBool.html#method.fetch_not), [`AtomicPtr::fetch_*`](https://doc.rust-lang.org/std/sync/atomic/struct.AtomicPtr.html#method.fetch_and), and [stronger CAS failure ordering](https://github.com/rust-lang/rust/pull/98383) available on Rust 1.34+.
- Provide workaround for bugs in the standard library's atomic-related APIs, such as [rust-lang/compiler-builtins#1234], [rust-lang/rust#100650], `fence`/`compiler_fence` on MSP430 that cause LLVM error, etc.

<!-- TODO:
- mention Atomic{I,U}*::fetch_neg, Atomic{I*,U*,Ptr}::bit_*, etc.
- mention optimizations not available in the standard library's equivalents
-->

portable-atomic version of `std::sync::Arc` and `std::task::Wake` is provided by the [portable-atomic-util] crate.

## Usage

Add this to your `Cargo.toml`:

```toml
[dependencies]
portable-atomic = "1"
```

The default features are mainly for users who use atomics larger than the pointer width.
If you don't need them, disabling the default features may reduce code size and compile time slightly.

```toml
[dependencies]
portable-atomic = { version = "1", default-features = false }
```

If your crate supports no-std environment and requires atomic CAS, enabling the `require-cas` feature will allow the portable-atomic to display a [helpful error message](https://github.com/taiki-e/portable-atomic/pull/100) to users on targets requiring additional action on the user side to provide atomic CAS.

```toml
[dependencies]
portable-atomic = { version = "1.3", default-features = false, features = ["require-cas"] }
```

(Since 1.8, portable-atomic can display a [helpful error message](https://github.com/taiki-e/portable-atomic/pull/181) even without the `require-cas` feature when the rustc version is 1.78+. However, the `require-cas` feature also allows rejecting builds at an earlier stage, we recommend enabling it unless enabling it causes [problems](https://github.com/matklad/once_cell/pull/267).)

## 128-bit atomics support

Native 128-bit atomic operations are available on x86_64 (Rust 1.59+), AArch64 (Rust 1.59+), riscv64 (Rust 1.59+), Arm64EC (Rust 1.84+), s390x (Rust 1.84+), and powerpc64 (Rust 1.95+), otherwise the fallback implementation is used.

On x86_64, even if `cmpxchg16b` is not available at compile-time (Note: `cmpxchg16b` target feature is enabled by default only on Apple, Windows (except Windows 7), and Fuchsia targets), run-time detection checks whether `cmpxchg16b` is available. If `cmpxchg16b` is not available at either compile-time or run-time detection, the fallback implementation is used. See also [`portable_atomic_no_outline_atomics`](#optional-cfg-no-outline-atomics) cfg.

They are usually implemented using inline assembly, and when using Miri or ThreadSanitizer that do not support inline assembly, core intrinsics are used instead of inline assembly if possible.

See the [`atomic128` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/atomic128/README.md) for details.

## <a name="optional-features"></a><a name="optional-cfg"></a>Optional features/cfgs

portable-atomic provides features and cfgs to allow enabling specific APIs and customizing its behavior.

Some options have both a feature and a cfg. When both exist, it indicates that the feature does not follow Cargo's recommendation that [features should be additive](https://doc.rust-lang.org/nightly/cargo/reference/features.html#feature-unification). Therefore, the maintainer's recommendation is to use cfg instead of feature. However, in the embedded ecosystem, it is very common to use features in such places, so these options provide both so you can choose based on your preference.

<details>
<summary>How to enable cfg (click to show)</summary>

One of the ways to enable cfg is to set [rustflags in the cargo config](https://doc.rust-lang.org/cargo/reference/config.html#targettriplerustflags):

```toml
# .cargo/config.toml
[target.<target>]
rustflags = ["--cfg", "portable_atomic_unsafe_assume_single_core"]
```

Or set environment variable:

```sh
RUSTFLAGS="--cfg portable_atomic_unsafe_assume_single_core" cargo ...
```

</details>

- <a name="optional-features-fallback"></a>**`fallback` feature** *(enabled by default)*<br>
  Enable fallback implementations.

  This enables atomic types with larger than the width supported by atomic instructions available on the current target. If the current target [supports 128-bit atomics](#128-bit-atomics-support), this is no-op.

  This uses fallback implementation that using global locks by default. The following features/cfgs change this behavior:
  - [`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg](#optional-features-unsafe-assume-single-core): Use fallback implementations that disabling interrupts instead of using global locks.
    - If your target is single-core and calling interrupt disable instructions is safe, this is a safer and more efficient option.
  - [`unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg](#optional-features-unsafe-assume-privileged): Use fallback implementations that using global locks with disabling interrupts.
    - If your target is multi-core and calling interrupt disable instructions is safe, this is a safer option.

- <a name="optional-features-float"></a>**`float` feature**<br>
  Provide `AtomicF{32,64}`.

  If you want atomic types for unstable float types ([`f16` and `f128`](https://github.com/rust-lang/rust/issues/116909)), enable unstable cfg (`portable_atomic_unstable_f16` cfg for `AtomicF16`, `portable_atomic_unstable_f128` cfg for `AtomicF128`, [there is no possibility that both feature and cfg will be provided for unstable options.](https://github.com/taiki-e/portable-atomic/pull/200#issuecomment-2682252991)).

<div class="rustdoc-alert rustdoc-alert-note">

> **ⓘ Note**
>
> - Atomic float's `fetch_{add,sub,min,max}` are usually implemented using CAS loops, which can be slower than equivalent operations of atomic integers. As an exception, AArch64 with FEAT_LSFE and GPU targets have atomic float instructions and we use them on AArch64 when `lsfe` target feature is available at compile-time. We [plan to use atomic float instructions for GPU targets as well in the future.](https://github.com/taiki-e/portable-atomic/issues/34)
> - Unstable cfgs are outside of the normal semver guarantees and minor or patch versions of portable-atomic may make breaking changes to them at any time.

</div>

- <a name="optional-features-std"></a>**`std` feature**<br>
  Use `std`.

- <a name="optional-features-require-cas"></a>**`require-cas` feature**<br>
  Emit compile error if atomic CAS is not available. See [Usage](#usage) section for usage of this feature.

- <a name="optional-features-serde"></a>**`serde` feature**<br>
  Implement `serde::{Serialize,Deserialize}` for atomic types.

  Note:
  - The MSRV when this feature is enabled depends on the MSRV of [serde].

- <a name="optional-features-critical-section"></a>**`critical-section` feature**<br>
  Use [critical-section] to provide atomic CAS for targets where atomic CAS is not available in the standard library.

  `critical-section` support is useful to get atomic CAS when the [`unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg)](#optional-features-unsafe-assume-single-core) can't be used,
  such as multi-core targets, unprivileged code running under some RTOS, or environments where disabling interrupts
  needs extra care due to e.g. real-time requirements.

<div class="rustdoc-alert rustdoc-alert-note">

> **ⓘ Note**
>
> - When enabling this feature, you should provide a suitable critical section implementation for the current target, see the [critical-section] documentation for details on how to do so.
> - With this feature, critical sections are taken for all atomic operations, while with `unsafe-assume-single-core` feature [some operations](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md#no-disable-interrupts) don't require disabling interrupts. Therefore, for better performance, if all the `critical-section` implementation for your target does is disable interrupts, prefer using `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) instead.
> - It is usually **discouraged** to always enable this feature in libraries that depend on `portable-atomic`.
>
>   Enabling this feature will prevent the end user from having the chance to take advantage of other (potentially) efficient implementations (implementations provided by `unsafe-assume-single-core` feature mentioned above, implementation proposed in [#60], etc.). Also, targets that are currently unsupported may be supported in the future.
>
>   The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature. (However, it may make sense to enable this feature by default for libraries specific to a platform where other implementations are known not to work.)
>
>   See also [this comment](https://github.com/matklad/once_cell/issues/264#issuecomment-2352654806).
>
>   As an example, the end-user's `Cargo.toml` that uses a crate that provides a critical-section implementation and a crate that depends on portable-atomic as an option would be expected to look like this:
>
>   ```toml
>   [dependencies]
>   portable-atomic = { version = "1", default-features = false, features = ["critical-section"] }
>   crate-provides-critical-section-impl = "..."
>   crate-uses-portable-atomic-as-feature = { version = "...", features = ["portable-atomic"] }
>   ```
>
> - Enabling both this feature and `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) will result in a compile error.
> - Enabling both this feature and `unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg) will result in a compile error.
> - The MSRV when this feature is enabled depends on the MSRV of [critical-section].

</div>

- <a name="optional-features-unsafe-assume-single-core"></a><a name="optional-cfg-unsafe-assume-single-core"></a>**`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg**<br>
  Assume that the target is single-core and privileged instructions required to disable interrupts are available.

  - When this feature/cfg is enabled, this crate provides atomic CAS for targets where atomic CAS is not available in the standard library by disabling interrupts.
  - When both this feature/cfg and enabled-by-default `fallback` feature is enabled, this crate provides atomic types with larger than the width supported by native instructions by disabling interrupts.

<div class="rustdoc-alert rustdoc-alert-warning">

> **⚠ Warning**
>
> This feature/cfg is `unsafe`, and note the following safety requirements:
> - Enabling this feature/cfg for multi-core systems is always **unsound**.
>
> - This uses privileged instructions to disable interrupts, so it usually doesn't work on unprivileged mode.
>
>   Enabling this feature/cfg in an environment where privileged instructions are not available, or if the instructions used are not sufficient to disable interrupts in the system, it is also usually considered **unsound**, although the details are system-dependent.
>
>   The following are known cases:
>   - On Arm (except for M-Profile architectures), this disables only IRQs by default. For many systems (e.g., GBA) this is enough. If the system need to disable both IRQs and FIQs, you need to enable the `disable-fiq` feature (or `portable_atomic_disable_fiq` cfg) together.
>   - On RISC-V, this generates code for machine-mode (M-mode) by default. If you enable the `s-mode` feature (or `portable_atomic_s_mode` cfg) together, this generates code for supervisor-mode (S-mode). In particular, `qemu-system-riscv*` uses [OpenSBI](https://github.com/riscv-software-src/opensbi) as the default firmware.

</div>

Consider using the [`unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg)](#optional-features-unsafe-assume-privileged) for multi-core systems with atomic CAS.

Consider using the [`critical-section` feature](#optional-features-critical-section) for systems that cannot use this feature/cfg.

See also the [`interrupt` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/interrupt/README.md).

<div class="rustdoc-alert rustdoc-alert-note">

> **ⓘ Note**
>
> - It is **very strongly discouraged** to enable this feature/cfg in libraries that depend on `portable-atomic`.
>
>   The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature/cfg. (However, it may make sense to enable this feature/cfg by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer targeting a single-core chip.)
> - Enabling this feature/cfg for unsupported architectures will result in a compile error.
>   - Arm, RISC-V, and Xtensa are currently supported. (Since all MSP430 and AVR are single-core, we always provide atomic CAS for them without this feature/cfg.)
>   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target is not supported yet.
> - Enabling this feature/cfg for targets where privileged instructions are obviously unavailable (e.g., Linux) will result in a compile error.
>   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target supports privileged instructions but the build rejected.
> - Enabling both this feature/cfg and `critical-section` feature will result in a compile error.
> - When both this feature/cfg and `unsafe-assume-privileged` feature (or `portable_atomic_unsafe_assume_privileged` cfg) are enabled, this feature/cfg is preferred.

</div>

- <a name="optional-features-unsafe-assume-privileged"></a><a name="optional-cfg-unsafe-assume-privileged"></a>**`unsafe-assume-privileged` feature / `portable_atomic_unsafe_assume_privileged` cfg**<br>
  Similar to `unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg, but only assumes about availability of privileged instructions required to disable interrupts.

  - When both this feature/cfg and enabled-by-default `fallback` feature is enabled, this crate provides atomic types with larger than the width supported by native instructions by using global locks with disabling interrupts.

<div class="rustdoc-alert rustdoc-alert-warning">

> **⚠ Warning**
>
> This feature/cfg is `unsafe`, and except for being sound in multi-core systems, this has the same safety requirements as [`unsafe-assume-single-core` feature / `portable_atomic_unsafe_assume_single_core` cfg](#optional-features-unsafe-assume-single-core).

</div>

<div class="rustdoc-alert rustdoc-alert-note">

> **ⓘ Note**
>
> - It is **very strongly discouraged** to enable this feature/cfg in libraries that depend on `portable-atomic`.
>
>   The recommended approach for libraries is to leave it up to the end user whether or not to enable this feature/cfg. (However, it may make sense to enable this feature/cfg by default for libraries specific to a platform where it is guaranteed to always be sound, for example in a hardware abstraction layer.)
> - Enabling this feature/cfg for unsupported targets will result in a compile error.
>   - This requires atomic CAS (`cfg(target_has_atomic = "ptr")` or `cfg_no_atomic_cas!`).
>   - Arm, RISC-V, and Xtensa are currently supported.
>   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target is not supported yet.
> - Enabling this feature/cfg for targets where privileged instructions are obviously unavailable (e.g., Linux) will result in a compile error.
>   - Feel free to [submit an issue](https://github.com/taiki-e/portable-atomic/issues/new) if your target supports privileged instructions but the build rejected.
> - Enabling both this feature/cfg and `critical-section` feature will result in a compile error.
> - When both this feature/cfg and `unsafe-assume-single-core` feature (or `portable_atomic_unsafe_assume_single_core` cfg) are enabled, `unsafe-assume-single-core` is preferred.

</div>

- <a name="optional-cfg-no-outline-atomics"></a>**`portable_atomic_no_outline_atomics` cfg**<br>
  Disable dynamic dispatching by run-time CPU feature detection.

  Dynamic dispatching by run-time CPU feature detection allows maintaining support for older CPUs while using features that are not supported on older CPUs, such as CMPXCHG16B (x86_64) and FEAT_LSE/FEAT_LSE2 (AArch64).

  See also the [`atomic128` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/atomic128/README.md).

<div class="rustdoc-alert rustdoc-alert-note">

> **ⓘ Note**
>
> - If the required target features are enabled at compile-time, dynamic dispatching is automatically disabled and the atomic operations are inlined.
> - This is compatible with no-std (as with all features except `std`).
> - On some targets, run-time detection is disabled by default mainly for compatibility with incomplete build environments or support for it is experimental, and can be enabled by `portable_atomic_outline_atomics` cfg. (When both cfg are enabled, `*_no_*` cfg is preferred.)
> - Some AArch64 targets enable LLVM's `outline-atomics` target feature by default, so if you set this cfg, you may want to disable that as well. (However, portable-atomic's outline-atomics does not depend on the compiler-rt symbols, so even if you need to disable LLVM's outline-atomics, you may not need to disable portable-atomic's outline-atomics.)
> - Dynamic detection is currently only supported in x86_64, AArch64, Arm, RISC-V, Arm64EC, and powerpc64. Enabling this cfg for unsupported architectures will be ignored.

</div>

## Related Projects

- [portable-atomic-util]: Synchronization primitives built with portable-atomic. This provides portable-atomic version of `std::sync::Arc` and `std::task::Wake`.
- [atomic-maybe-uninit]: Atomic operations on potentially uninitialized integers.
- [atomic-memcpy]: Byte-wise atomic memcpy.
- [asmtest]: A library for tracking generated assemblies.

[#60]: https://github.com/taiki-e/portable-atomic/issues/60
[asmtest]: https://github.com/taiki-e/asmtest
[atomic-maybe-uninit]: https://github.com/taiki-e/atomic-maybe-uninit
[atomic-memcpy]: https://github.com/taiki-e/atomic-memcpy
[critical-section]: https://github.com/rust-embedded/critical-section
[portable-atomic-util]: https://github.com/taiki-e/portable-atomic-util
[rust-lang/compiler-builtins#1234]: https://github.com/rust-lang/compiler-builtins/pull/1234
[rust-lang/rust#100650]: https://github.com/rust-lang/rust/issues/100650
[serde]: https://github.com/serde-rs/serde

<!-- tidy:sync-markdown-to-rustdoc:end -->
*/


Source: src/tests/helper.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/tests/mod.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: src/utils.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


Source: version.rs:1
// SPDX-License-Identifier: Apache-2.0 OR MIT


