Metadata-Version: 2.4
Name: bigbt
Version: 1.0.0
Summary: Big Data Analytics lab experiments (HDFS, Sqoop, HBase, Pig, Hive, MongoDB, Bloom filter, Flajolet-Martin, R) as ready-to-run bash scripts
License-Expression: MIT
Keywords: big data,hadoop,hdfs,hive,pig,hbase,sqoop,mongodb,bloom filter,flajolet-martin,r,lab,practical
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Classifier: Environment :: Console
Classifier: Intended Audience :: Education
Classifier: Topic :: Education
Classifier: Topic :: Database
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# bigbt

Big Data Analytics lab experiments as ready-to-run bash scripts.

```bash
pip install bigbt
big experiments        # creates ./experiments with setup.sh, exp1.sh ... exp9.sh
cd experiments
bash setup.sh          # check installed tools, start HDFS + YARN
bash exp1.sh           # run experiment 1
```

Each `expN.sh` is self-contained: it writes the data, Pig / HiveQL / R / JavaScript files it
needs into `expN_files/`, then runs them, printing every command before its output.

## Experiments

| File | Experiment | Options |
|---|---|---|
| `setup.sh` | check tools, start HDFS + YARN | `check`, `stop` |
| `exp1.sh` | HDFS commands | `set1` … `set6`, `clean` |
| `exp2.sh` | Sqoop: MySQL ⇄ HDFS (edit `DB_USER` / `DB_PASS` at the top) | `setup`, `import`, `variants`, `export` |
| `exp3.sh` | HBase shell (+ Java client) | `clean`, `java` |
| `exp4.sh` | Pig Latin | `mr` (MapReduce mode on HDFS) |
| `exp5.sh` | Hive: database + descriptive analytics | `external` |
| `exp6.sh` | Bloom filter in R | `1` strings, `2` integer hashes |
| `exp7.sh` | Flajolet-Martin algorithm in R | `1`, `2` |
| `exp8.sh` | MongoDB | `paste` |
| `exp9.sh` | R programming, questions 1-15 | `1` … `15`, `list` |

Pass an option as the first argument, e.g. `bash exp1.sh set3` or `bash exp9.sh 5`.

## Commands

```
big experiments [-o DIR] [-f]   create the folder (default ./experiments; -f overwrites)
big list                        one-line description of every script
big show N                      print expN.sh to the terminal
```

## Requirements

Python 3.8+ to install. The scripts need the tools of each experiment on `PATH`
(Hadoop, Sqoop + MySQL, HBase, Pig, Hive or beeline, `mongosh`, R).

`experiments/data/*_sample.csv` are synthetic practice files with the column layout of the
churn, supermarket-sales and loan datasets; put the real CSV in `exp9_files/` to use it instead.
