Metadata-Version: 2.4 Name: EfficientWord-Net Version: 1.0.3 Summary: Few Shot Learning based Hotword Detection Engine License: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2021 ANT-BRaiN Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. Project-URL: Homepage, https://github.com/Ant-Brain/EfficientWord Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Requires-Python: <3.15,>=3.10 Description-Content-Type: text/markdown License-File: LICENSE.md Requires-Dist: numpy Requires-Dist: onnxruntime Requires-Dist: pyaudio Requires-Dist: pydub>=0.25.1 Requires-Dist: requests Requires-Dist: typer>=0.21.0 Dynamic: license-file # EfficientWord-Net: Hotword Detection Based on Few-Shot Learning Home assistants require special phrases called hotwords to get activated (e.g., "OK Google"). EfficientWord-Net is a hotword detection engine based on few-shot learning that allows developers to add custom hotwords to their programs without extra charges. The library is purely written in Python and uses Google's TFLite implementation for faster real-time inference. It is inspired by FaceNet's Siamese Network Architecture and performs best when 3-4 hotword samples are collected directly from the user. ### Demo of EfficientWord-Net on Pi ## Access Training File [Training File](./training.ipynb) to access the training file. ## Datasets Here are the links: - [Dataset 1](https://drive.google.com/file/d/1f6dp72D9WxErXvaZP6KIBLv4-eKpvLZa/view?usp=sharing) - [Dataset 2](https://drive.google.com/file/d/19QUTiAZvF1pFy5BeaV_rc83MeDu8yojv/view?usp=sharing) ## Access Paper [Research Paper](https://worldscientific.com/doi/10.1142/S0219649222500599) to access the research paper. ## Python Version Requirements This library works with Python versions 3.10 to 3.14. ## Dependencies Installation Before running the pip installation command for the library, a few dependencies need to be installed manually: - [PyAudio (depends on PortAudio)](https://abhgog.gitbooks.io/pyaudio-manual/content/installation.html) ## Package Installation Run the following pip command: ``` pip install EfficientWord-Net ``` To import the package: ```python import eff_word_net ``` ## Demo After installing the packages, you can run the demo script built into the library (ensure you have a working microphone). Access documentation from: Command to run the demo: ``` python -m eff_word_net.engine ``` ## Generating Custom Wakewords For any new hotword, the library needs information about the hotword. This information is obtained from a file called `{wakeword}_ref.json`. For example, for the wakeword 'alexa', the library would need the file called `alexa_ref.json`. These files can be generated with the following procedure: 1. Collect 4 to 10 uniquely sounding pronunciations of a given wakeword. Put them into a separate folder that doesn't contain anything else. 2. Finally, run this command. It will ask for the input folder's location (containing the audio files) and the output folder (where the _ref.json file will be stored): ``` python -m eff_word_net.generate_reference ``` The pathname of the generated wakeword needs to be passed to the HotwordDetector instance: ```python HotwordDetector( hotword="hello", model=Resnet_50_Arc_loss(), reference_file="/full/path/name/of/hello_ref.json", threshold=0.9, # min confidence required to consider a trigger relaxation_time=0.8 # default value, in seconds ) ``` The model variable can receive an instance of Resnet_50_Arc_loss or First_Iteration_Siamese. The relaxation_time parameter is used to determine the minimum time between any two triggers. Any potential triggers before the relaxation_time will be canceled. The detector operates on a sliding window approach, resulting in multiple triggers for a single utterance of a hotword. The relaxation_time parameter can be used to control multiple triggers; in most cases, 0.8 seconds (default) will suffice. ## Out-of-the-Box Sample Hotwords The library has predefined embeddings readily available for a few wakewords such as **Mycroft**, **Google**, **Firefox**, **Alexa**, **Mobile**, and **Siri**. Their paths are readily available in the library installation directory. ```python from eff_word_net import samples_loc ```
## Try your first single hotword detection script ```python import os from eff_word_net.streams import SimpleMicStream from eff_word_net.engine import HotwordDetector from eff_word_net.audio_processing import Resnet50_Arc_loss from eff_word_net import samples_loc base_model = Resnet50_Arc_loss() mycroft_hw = HotwordDetector( hotword="mycroft", model = base_model, reference_file=os.path.join(samples_loc, "mycroft_ref.json"), threshold=0.7, relaxation_time=2 ) mic_stream = SimpleMicStream( window_length_secs=1.5, sliding_window_secs=0.75, ) mic_stream.start_stream() print("Say Mycroft ") while True : frame = mic_stream.getFrame() result = mycroft_hw.scoreFrame(frame) if result==None : #no voice activity continue if(result["match"]): print("Wakeword uttered",result["confidence"]) ```
## Detecting Mulitple Hotwords from audio streams The library provides a computation friendly way to detect multiple hotwords from a given stream, instead of running `scoreFrame()` of each wakeword individually ```python import os from eff_word_net.streams import SimpleMicStream from eff_word_net import samples_loc print(samples_loc) base_model = Resnet50_Arc_loss() mycroft_hw = HotwordDetector( hotword="mycroft", model = base_model, reference_file=os.path.join(samples_loc,"mycroft_ref.json"), threshold=0.7, relaxation_time=2 ) alexa_hw = HotwordDetector( hotword="alexa", model=base_model, reference_file=os.path.join(samples_loc,"alexa_ref.json"), threshold=0.7, relaxation_time=2, #verbose=True ) computer_hw = HotwordDetector( hotword="computer", model=base_model, reference_file=os.path.join(samples_loc,"computer_ref.json"), threshold=0.7, relaxation_time=2, #verbose=True ) multi_hotword_detector = MultiHotwordDetector( [mycroft_hw, alexa_hw, computer_hw], model=base_model, continuous=True, ) mic_stream = SimpleMicStream(window_length_secs=1.5, sliding_window_secs=0.75) mic_stream.start_stream() print("Say ", " / ".join([x.hotword for x in multi_hotword_detector.detector_collection])) while True : frame = mic_stream.getFrame() result = multi_hotword_detector.findBestMatch(frame) if(None not in result): print(result[0],f",Confidence {result[1]:0.4f}") ```
Access documentation of the library from here : Here's the corrected version of the README.md file with improved grammar and formatting: ## Change notes from v1.0.2 to v1.0.4 - Depreceated tensorflow model due to inference difficulties of tflite in platforms like Windows ## Change notes from 0.2.2 to v1.0.1 ### New Model Addition: Resnet_50_Arc_loss with huge improvements - Trained a new model from scratch using a modified distilled dataset from MLCommons. - Used Arc loss function instead of triplet loss function. - The resultant model is stored as resnet_50_arcloss. - The newer model showcases much better resilience towards background noise and requires fewer samples for good accuracy. - Minor changes in the API flow to facilitate easy addition of newer models. - Newer model can handle a fixed window length of 1.5 seconds. - The old model can still be accessed through first_iteration_siamese. ## Change notes from v0.1.1 to 0.2.2 - Major changes to replace complex logic of handling poly triggers per utterance with simpler logic and a more straightforward API for programmers. - Introduces breaking changes. - C++ implementation of the current model is [here](https://github.com/Ant-Brain/EfficientWord-Net/issues/56). ## Limitations in Current Model - Trained on single words, hence may result in bizarre behavior when using phrases like "Hey xxx". - Audio processing window limited to 1 sec. Hence, it will not work effectively for longer hotwords. ## FAQ - **Hotword Performance is bad**: If you are experiencing issues like this, feel free to ask in the [discussions](https://github.com/Ant-Brain/EfficientWord-Net/discussions/4). - **Can it run on FPGAs like Arduino?**: No, the new Resnet_50_Arcloss model is too heavy to run on Arduino (roughly 88MB in size). We will soon add support for pruned versions of the model so that it can become light enough to run on tiny devices. For now, it should be able to run on Raspberry Pi-like devices. ## Contribution - If you have ideas to make the project better, feel free to ping us in the [discussions](https://github.com/Ant-Brain/EfficientWord-Net/discussions/3). - The current [logmelcalc.tflite](/eff_word_net/logmelcalc.tflite) graph can convert only 1 audio frame to Log Mel Spectrogram at a time. It would be of great help if TensorFlow gurus out there could assist us with this. ## TODO - Add audio file handler in streams. PRs are welcome. - Remove librosa requirement to encourage generating reference files directly on edge devices. - Add more detailed documentation explaining the sliding window concept. - Add model fine-tuning support. - Add support for sparse and fine-grained pruning where the resultant models could be used for fine-tuning (already working on this). ## Support Us Our hotword detector's performance is notably lower compared to Porcupine. We have thought about better NN architectures for the engine and hope to outperform Porcupine. This has been our undergrad project, so your support and encouragement will motivate us to develop the engine further. If you love this project, recommend it to your peers, give us a 🌟 on GitHub, and a clap 👏 on [Medium](https://link.medium.com/yMBmWGM03kb). Update: Your stars encouraged us to create a new model which is far better. Let's make this community grow! ## License [Apache License 2.0](/LICENSE.md)