# dbprocessing

# dbprocessing.DButils.DButils

class dbprocessing.DButils.DButils(mission='Test', db_var=None, echo=False, engine=None)

  Utility routines for DBProcessing class

  All of these may be user called but are meant to be internal routines for DBProcessing

  Warning

  It is strongly encouraged to make sure the database is closed before the program terminates, either by calling `closeDB()` or deleting instances of this object (with an explicit del or by allowing it to go out of scope.) If this object still exists at interpreter exit, it will attempt to close the database, but the functionality to do so may have already been torn down. See for example Python issue 39513.

  __init__(mission='Test', db_var=None, echo=False, engine=None)

    Initialize the DButils class

    Parameters
      mission `str`

        Name of the mission. This may be the name of a .sqlite file or the name of a Postgresql database; see Specifying a database for Postgresql support (implemented by `postgresql_url()`).

      echo `bool`, default False

        if True, the Engine will log all statements as well as a repr() of their parameter lists to the logger

      engine `str`, optional

        DB engine to connect to (e.g sqlite, postgresql). Defaults to sqlite if mission is an existing file, else postgresql.

    Other Parameters
      db_var

        Does nothing

  Methods

  `ProcessqueueClean`([dryrun]) | Keep only latest version of each file in the process queue.
  `ProcessqueueFlush`() | remove everything from the process queue
  `ProcessqueueGet`([index, instance]) | Get the file at the head of the queue (from the left)
  `ProcessqueueGetAll`([version_bump]) | Return the entire contents of the process queue
  `ProcessqueueLen`() | Return the number of files in the process queue
  `ProcessqueuePop`([index]) | pop a file off the process queue (from the left)
  `ProcessqueuePush`(fileid[, version_bump, MAX_ADD]) | Push a file onto the process queue (onto the right)
  `ProcessqueueRawadd`(fileid[, version_bump, ...]) | raw add file ids to the process queue
  `ProcessqueueRemove`(item[, commit]) | remove a file from the queue by name or number
  `addCode`(filename, relative_path, ...[, ...]) | Add an executable code to the DB
  `addFile`([filename, data_level, version, ...]) | Add a datafile to the database.
  `addFilecodelink`(resulting_file_id, source_code) | Add a file code link to the database
  `addFilefilelink`(resulting_file_id, source_file) | Add a file file link to the database
  `addInspector`(filename, relative_path, ...[, ...]) | Add an inspector to the DB.
  `addInstrument`(instrument_name, satellite_id) | Add a Instrument to the database
  `addInstrumentproductlink`(instrument_id, ...) | Add a instrument product link to the database
  `addLogging`(currently_processing, ...[, pid, ...]) | Add an entry to the logging table
  `addMission`(mission_name, rootdir, incoming_dir) | Add a mission to the database
  `addProcess`(process_name, output_product, ...) | Add a process to the database
  `addProduct`(product_name, instrument_id, ...) | Add a product to the database
  `addRelease`(filename, release[, commit]) | Given a filename or file_id add an entry to the release table
  `addSatellite`(satellite_name, mission_id) | Add a satellite to the database
  `addUnixTimeTable`() | Add a table containing a file's Unix start/stop time.
  `addproductprocesslink`(input_product_id, ...) | Add a product process link to the database
  `checkDiskForFile`(file_id[, fix]) | Check if the file existence on disk matches database record
  `checkFileSHA`(file_id) | Given a file id or name check the db checksum and the file checksum
  `checkFiles`([limit]) | Check files in the DB, return inconsistent files and why
  `checkIncoming`([glb]) | Check the incoming directory for the current mission
  `closeDB`() | Close the database connection
  `codeIsActive`(ec_id, date) | Determine if a code is active and newest version.
  `commitDB`() | Do the commit to the DB
  `currentlyProcessing`() | Checks the db to see if it is currently processing
  `delFilecodelink`(f[, commit]) | Remove entries from Filecodelink for a Given file
  `delFilefilelink`(f[, commit]) | Remove entries from Filefilelink
  `delInspector`(i) | Removes an inspector from the db
  `delProduct`(pp) | Removes a product from the db
  `delProductProcessLink`(ll) | Removes a product from the db
  `editTable`(table, my_id, column[, my_str, ...]) | Apply string editing operations on a single row, column of a table
  `fileIsNewest`(filename[, debug]) | quesry the database, is this filename or file_id newest version?
  `file_id_Clean`(invals) | Given a list of file IDs return only newest versions of matching files.
  `getActiveInspectors`() | Query the db and returns all active inspectors
  `getAllCodes`([active]) | Return a list of all codes
  `getAllCodesFromProcess`(proc_id) | Given a process id return the code ids that performs that process
  `getAllFileIds`([fullPath, startDate, ...]) | Return all the file ids in the database All parameters are optional; if not specified, default is "all".
  `getAllFilenames`([fullPath, startDate, ...]) | Return all the file names in the database
  `getAllInstruments`() | Return dictionaries of instrument traceback dictionaries
  `getAllProcesses`([timebase]) | Get all processes
  `getAllProducts`([id_only]) | Return a list of all products as instances
  `getAllSatellites`() | Return dictionaries of satellite, mission objects
  `getChildTree`(inprod) | Given an input product return a list of its output product ids
  `getChildrenProcesses`(file_id) | Given a file, return all the processes that use this as input
  `getCodeDirectory`() | Return the code directory for the current mission
  `getCodeFromProcess`(proc_id, utc_file_date) | Given a process id return the code id that performs that process on a particular date.
  `getCodeID`(codename) | Return the codeID for a code's filename.
  `getCodePath`(code_id) | Given a code_id list return the full name (path and all) of the code
  `getCodeVersion`(code_id) | Given a code_id get the code version Given a code_id list return the full name (path and all) of the code
  `getDirectory`(column[, default]) | Look up directory for the specified column.
  `getEntry`(table, args) | Return entry instance from any table in DB
  `getErrorPath`() | Return the error directory for the current mission
  `getFileDates`(file_id) | Given a file_id or name return the dates it spans
  `getFileFullPath`(filename) | Return the full path to a file given the name or id
  `getFileID`(filename) | Return the fileID for the input filename
  `getFileParents`(file_id[, id_only]) | Given a file_id (or filename) return the files that went into making it
  `getFileVersion`(fileid) | Return the version instance for a file
  `getFilecodelink_bycode`(code_id) | Given a code_id return the file_id of all files it created
  `getFilecodelink_byfile`(file_id) | Given a file_id return the code_id that created it, or None
  `getFiles`([startDate, endDate, level, ...]) | Query database for file records, with filters.
  `getFilesByCode`(code_id[, newest_version, ...]) | Given a code_id (or name) return the files that were created using it
  `getFilesByDate`(daterange[, newest_version]) | Return files in the db with utc_file_date in the range specified
  `getFilesByInstrument`(inst_id[, level, ...]) | Given an instrument_id return all the file instances associated with it
  `getFilesByProduct`(prod_id[, newest_version]) | Given a product_id or name return all the files associated with it
  `getFilesByProductDate`(product_id, daterange) | Return the files by product id with utc_file_date in range specified
  `getFilesByProductTime`(product_id, daterange) | Return the files in the db by product id with any data in date range
  `getIncomingPath`() | Return the incoming directory for the current mission
  `getInputProductID`(process_id[, range]) | Return the input products for a particular process.
  `getInspectorDirectory`() | Return the inspector directory for the current mission
  `getInstrumentID`(name[, satellite_id]) | Return the instrument_id for a given instrument.
  `getMissionDirectory`() | Return the root directory for the current mission
  `getMissionID`(mission_name) | Given a mission name return its ID
  `getMissions`() | Return a list of all the missions
  `getProcessFromInputProduct`(product) | Given a product id return all the processes that use that as an input
  `getProcessFromOutputProduct`(outProd) | Gets process from the db that have the output product
  `getProcessID`(proc_name) | Given a process name return its id
  `getProcessTimebase`(process_id) | Return the timebase for a process
  `getProductID`(product_name) | Return the product ID for an input product name
  `getProductParentTree`() | go through the db and return a tree of all products and their parents
  `getProductsByInstrument`(inst_id) | Get all the products for a Given instrument
  `getProductsByLevel`(level) | Get all the products for a Given level
  `getRunProcess`() | Return a list of the processes who's output_timebase is "RUN"
  `getSatelliteID`(sat_name) | Returns the satellite ID for an input satellite name
  `getSatelliteMission`(sat_name) | Given a satellite or satellite id return the mission
  `getTraceback`(table, in_id[, in_id2]) | Master routine for all the getXXXTraceback functions
  `list_release`(rel_num[, fullpath]) | Given a release number return all the filenames with the release
  `openDB`(engine[, db_var, verbose, echo]) | Setup python to talk to the database
  `purgeProcess`(proc[, commit]) | Remove process and productprocesslink
  `renameFile`(filename, newname) | Rename a file in the db
  `resetProcessingFlag`(comment) | Query the db and reset a processing flag
  `startLogging`() | Add an entry to the logging table in the DB, logging
  `stopLogging`(comment) | Finish the entry to the processing table in the DB, logging
  `tag_release`(rel_num) | Tag all the newest versions of files to a release number (integer)
  `updateCodeNewestVersion`(code_id[, is_newest]) | Update a code to indicate whether it's the newest version.
  `updateInspectorSubs`(insp_id) | Update an existing inspector performing the {} replacements
  `updateProcessSubs`(proc_id) | Update an existing product performing the {} replacements
  `updateProductSubs`(product_id) | Update an existing product performing the {} replacements

  ProcessqueueClean(dryrun=False)

    Keep only latest version of each file in the process queue.

    This is determined by `product_id` and `utc_file_date`. Also sorts queue by level, date

    Parameters
      dryrun `bool`, default False

        Do not actually make changes to the queue.

  ProcessqueueFlush()

    remove everything from the process queue

    This is as optimized as it can be

  ProcessqueueGet(index=0, instance=False)

    Get the file at the head of the queue (from the left)

    Returns
      file_id `int`

        the `file_id` of the `file` popped from the queue

  ProcessqueueGetAll(version_bump=False)

    Return the entire contents of the process queue

    Parameters
      version_bump `bool`, default False

        Include the version bump information

    Returns
      `list`

        All `file_id` in the process queue, optionally the `version_bump` information as well.

  ProcessqueueLen()

    Return the number of files in the process queue

    Returns
      `int`

        Count of files in the queue

  ProcessqueuePop(index=0)

    pop a file off the process queue (from the left)

    Returns
      file_id `int`

        the `file_id` of the `file` popped from the queue

    Other Parameters
      index `int`

        the index in the queue to pop

  ProcessqueuePush(fileid, version_bump=None, MAX_ADD=150)

    Push a file onto the process queue (onto the right)

    Parameters
      fileid `int` or `str`

        the `file_id` or `filename` to put on the process queue.

    Returns
      file_id `int`

        `file_id` of the file placed on queue, as grabbed from the db.

  ProcessqueueRawadd(fileid, version_bump=None, commit=True)

    raw add file ids to the process queue

    Warning

    This might break things if an id is added that does not exist; it’s meant to be fast and used after getting the ids. IS safe against adding ids that are already in the queue.

    Parameters
      fileid `int` or `Iterable`

        the `file_id` or sequence of file ids to add

    Returns
      num `int`

        the number of entries added to the processqueue

  ProcessqueueRemove(item, commit=True)

    remove a file from the queue by name or number

    Parameters
      item `int` or `str`

        `filename` or `file_id` of file to remove from the process queue.

      commit `bool`, default True

        Commit changes to the database when done.

  addCode(filename, relative_path, code_start_date, code_stop_date, code_description, process_id, version, active_code, date_written, output_interface_version, newest_version, arguments=None, cpu=1, ram=1)

    Add an executable code to the DB

    Creates a record in `code` table.

    Parameters
      filename `str`

        the filename of the code.

      relative_path `str`

        the relative path (relative to mission code directory).

      code_start_date `datetime`

        start of validity of the code.

      code_stop_date `datetime`

        end of validity of the code.

      code_description `str`

        description of the code (50 char).

      process_id `int`

        `process_id` of the process this code implements.

      version `Version` or `str`

        Version of the code.

      active_code `bool`

        if the code is active.

      code_date_written `datetime`

        date the code was written.

      output_interface_version `int`

        Interface version of files produced by the code.

      newest_version `bool`

        Is the code the newest version.

      arguments `str`, optional

        Additional command line arguments to pass to the code, default none (no extra arguments).

      cpu `int`, default 1

        Relative CPU usage of code (usually in terms of threads).

      ram `float`, default 1

        Relative memory usage of code.

    Returns
      code_id `int`

        `code_id` of newly created record.

  addFile(filename=None, data_level=None, version=None, file_create_date=None, exists_on_disk=None, utc_file_date=None, utc_start_time=None, utc_stop_time=None, check_date=None, verbose_provenance=None, quality_comment=None, caveats=None, met_start_time=None, met_stop_time=None, product_id=None, shasum=None, process_keywords=None, quality_checked=None)

    Add a datafile to the database.

    Adds record to `file`.

    Parameters
      filename `str`

        Filename to add.

      data_level `float`

        The data level of the file.

      version `Version`

        The version of the file to create.

      file_create_date `datetime`

        Date and time the file was created.

      exists_on_disk `bool`

        Does the file exist on disk.

      product_id `int`

        `product_id` of the product the file belongs to.

      utc_file_date `date`

        The UTC date of the file.

      utc_start_time `datetime`

        UTC of first timestamp in file.

      utc_end_time `datetime`

        UTC of last timestamp in file.

      check_date `datetime`

        The date the file was quality checked.

      verbose_provenance `str`

        Verbose provenance of the file.

      quality_comment `str`

        Comment on quality from quality check.

      caveats `str`

        Caveats on use of file.

      met_start_time `int`

        MET of first timestamp in file.

      met_stop_time `int`

        MET of last timestamp in file.

    Returns
      `int`

        `file_id` of the newly inserted file record.

    Notes

    All arguments are technically optional, but the insertion to the database may fail if an argument is not provided for a column which requires a non-NULL value. See `file`.

  addFilecodelink(resulting_file_id, source_code)

    Add a file code link to the database

    Connects file to code that made it via `filecodelink`.

    Parameters
      resulting_file_id `int`

        `file_id` of the created file

      source_code `int`

        `code_id` of the code that created the file

  addFilefilelink(resulting_file_id, source_file)

    Add a file file link to the database

    Links a file to one of its input files via `filefilelink`.

    Parameters
      resulting_file_id `int`

        `file_id` of the output file.

      source_file `int`

        `file_id` of the input file.

  addInspector(filename, relative_path, description, version, active_code, date_written, output_interface_version, newest_version, product, arguments=None)

    Add an inspector to the DB.

    Creates a record in `inspector` table.

    Parameters
      filename `str`

        the filename of the inspector

      relative_path `str`

        the relative path (relative to mission inspector directory).

      description `str`

        description of the inspector (50 char).

      version `Version` or `str`

        Version of the code.

      active_code `bool`

        if the inspector is active.

      date_written `datetime`

        date the inspector was written.

      output_interface_version `int`

        Written to database, but not used.

      newest_version `bool`

        Is the inspector the newest version.

      product `int`

        `product_id` of the product this inspector identifies.

      arguments `str`, optional

        Additional keywords to pass to the `inspect()` method, default none (no extra arguments).

    Returns
      inspector_id `int`

        `inspector_id` of newly created record.

  addInstrument(instrument_name, satellite_id)

    Add a Instrument to the database

    Creates record in `instrument`.

    Parameters
      instrument_name `str`

        The name of the instrument (`instrument_name`).

      satellite_id `int`

        `satellite_id` of the satellite associated with the instrument.

  addInstrumentproductlink(instrument_id, product_id)

    Add a instrument product link to the database

    Links a product to its instrument via `instrumentproductlink`.

    Parameters
      instrument_id `int`

        `instrument_id` of the instrument.

      product_id `int`

        `product_id` of the product.

  addLogging(currently_processing, processing_start_time, mission_id, user, hostname, pid=None, processing_end_time=None, comment=None)

    Add an entry to the logging table

    Parameters
      currently_processing `bool`

        is the db currently processing?

      processing_start_time `datetime`

        the time the processing started

      mission_id `int`

        the `mission_id` the processing is for

      user `str`

        the user doing the processing

      hostname `str`

        the hostname that initiated the processing

      pid `int`, optional

        the process id that did the processing, default null

      processing_end_time `datetime`, optional

        the time the processing stopped, default null

      comment `str`

        comment about the processing run

    Returns
      Logging

        instance of the class for the `logging` table.

  addMission(mission_name, rootdir, incoming_dir, codedir=None, inspectordir=None, errordir=None)

    Add a mission to the database

    Optional directories which are not specified will be inserted into the database as nulls, and the default will be determined at runtime.

    Parameters
      mission_name `str`

        the name of the mission

      rootdir `str`

        the root directory of the mission

      incoming_dir `str`

        directory for incoming files

      codedir `str`, optional

        directory containing codes; default, see `getCodeDirectory()`

      inspectordir `str`, optional

        directory containing product inspectors; default, see `getInspectorDirectory()`)

      errordir `str`, optional

        directory to contain error files; default, see `getErrorPath()`

  addProcess(process_name, output_product, output_timebase, extra_params=None, trigger=None)

    Add a process to the database

    Parameters
      process_name `str`

        the name of the process (`process_name`).

      output_product `int`

        the output product id (`output_product`).

      output_timebase `str`

        Timebase to use for output files, options `RUN`, `ORBIT`, `DAILY`, `WEEKLY`, `MONTHLY`, `YEARLY`, `FILE` (`output_timebase`).

      extra_params `str`, optional

        extra parameters to pass to the code (`extra_params`).

    Other Parameters
      trigger

        Unused.

  addProduct(product_name, instrument_id, relative_path, format, level, product_description)

    Add a product to the database

    Adds record to `product`.

    Parameters
      product_name `str`

        the name of the product

      instrument_id `int`

        the instrument the product is from

      relative_path `str`

        relative path for the product

      format `str`

        the format of the product filenames

  addRelease(filename, release, commit=False)

    Given a filename or file_id add an entry to the release table

    Parameters
      filename `int` or `str`

        `filename` or `file_id` of file to add to a release.

      release `int`

        Release number to add file to.

      commit `bool`, default False

        Commit changes to the database when done.

    See also

    Releases
  addSatellite(satellite_name, mission_id)

    Add a satellite to the database

    Parameters
      satellite_name `str`

        the name of the satellite

      mission_id `int`

        `mission.mission_id` of mission to add to

    Returns
      `int`

        `satellite.satellite_id` of newly-added satellite.

  addUnixTimeTable()

    Add a table containing a file’s Unix start/stop time.

    Used for migrating databases; doing file searches based on the Unix time is faster than the UTC timestamp. This will also populate the time columns from a file’s UTC start/stop time.

    Raises
      RuntimeError

        If the Unix time table already exists

  addproductprocesslink(input_product_id, process_id, optional, yesterday=0, tomorrow=0)

    Add a product process link to the database

    Connects input product to output via `productprocesslink`.

    Parameters
      input_product_id `int`

        `product_id` of the input product.

      process_id `int`

        `process.process_id` of the process for which `input_product_id` is an input.

      optional `bool`

        if the input product is optional (vs. required)

      yesterday `int`, default 0

        How many extra days back do you need

      tomorrow `int`, default 0

        How many extra days forward do you need

  checkDiskForFile(file_id, fix=False)

    Check if the file existence on disk matches database record

    Parameters
      file_id `int`

        `file_id` of the file to check

      fix `bool`, default False

        set to have the DB fixed to match the file system this is NOT sure to be safe

    Returns
      `bool`

        True if consistent, False otherwise

  checkFileSHA(file_id)

    Given a file id or name check the db checksum and the file checksum

    Parameters
      file_id `int` or `str`

        `filename` or `file_id` of file to check

    Returns
      `bool`

        If the calculated checksum of file on disk matches the checksum in the database.

  checkFiles(limit=None)

    Check files in the DB, return inconsistent files and why

    Parameters
      limit `int`

        Maximum number of files to check, default all

    Returns
      :class:`list` of class:tuple

        All files with problems. Each element is (`filename`, result), where “result” is 1 for a bad checksum and 2 if file not found on disk.

  checkIncoming(glb='*')

    Check the incoming directory for the current mission

    Parameters
      glb `str`, optional

        Glob pattern that files must match.

    Returns
      :class:`list` of class:str

        All files in the incoming directory

  closeDB()

    Close the database connection

    Examples

        >>>  pnl.closeDB()

  codeIsActive(ec_id, date)

    Determine if a code is active and newest version.

    Parameters
      ec_id `int` or `str`

        `code_id` or `code_description` of the code to check.

      date `date`

        Check if code is valid for files on this date (corresponds to `utc_file_date`).

    Returns
      `bool`

        If code is active, newest version, and `date` falls within the code’s valid date range.

  commitDB()

    Do the commit to the DB

  currentlyProcessing()

    Checks the db to see if it is currently processing

    Ensures not doing 2 at the same time

    Returns
      :class:`bool` or class:int

        False or the current process id

    Examples

        >>>  pnl.currentlyProcessing()

  delFilecodelink(f, commit=True)

    Remove entries from Filecodelink for a Given file

    Remove record from `filecodelink` if the file was created by a code.

    Parameters
      f `int` or `str`

        `file_id` or `filename` of file to unassociate with code.

      commit `bool`, default True

        Commit changes to the database when done.

  delFilefilelink(f, commit=True)

    Remove entries from Filefilelink

    Remove record from `filefilelink` if the file is in either `source_file` or `resulting_file`.

    Parameters
      f `int` or `str`

        `file_id` or `filename` of file to remove from link.

      commit `bool`, default True

        Commit changes to the database when done.

  delInspector(i)

    Removes an inspector from the db

    Parameters
      i `int`

        `inspector.inspector_id` of inspector to delete

  delProduct(pp)

    Removes a product from the db

    Parameters
      pp `int` or `str`

        `product_id` or `product_name` of product to remove.

  delProductProcessLink(ll)

    Removes a product from the db

    Parameters
      ll `list`

        Two elements, `process_id` and `input_product_id` of record to remove from `productprocesslink`.

    Notes

    Untested!

  editTable(table, my_id, column, my_str=None, after_flag=None, ins_after=None, ins_before=None, replace_str=None, combine=False)

    Apply string editing operations on a single row, column of a table

    For a specified row and column of a table, update the value according to operations specified by the combination of the kwargs.

    To replace all instances of a string with another, set `replace_str` to the string to replace and `my_str` to the new value to replace it with.

    To append a string to all instance of a string, set `ins_after` to the existing string and `my_str` to the value to append.

    To prepend a string to all instance of a string, set `ins_after` to the existing string and `my_str` to the value to prepend.

    When operating on the `arguments` column of the `code` table, and `after_flag` is specified, all three of these operations will only apply to the “word” (whitespace-separated) after the “word” in `after_flag`. See examples.

    When operating on the `arguments` column of the `code` table, `combine` may be set to `True` to combine every word that follows each instance of `after_flag` into a comma-separated list after a single instance of `after_flag`. See examples.

    One and only one of `ins_after`, `ins_before`, `replace_str` and `combine` can be specified; there is no default operation. If `ins_after`, `ins_before`, or `replace_str` are specified, `my_str` must be.

    Note

    Written and tested for code table. Not thoroughly tested for others.

    Parameters
      table `str`

        Name of the table to edit.

      my_id `int`

        Record to edit; most commonly the numerical ID (primary key) but also supports string matching on other columns as provided by `getEntry()`.

      column `str`

        name of column to edit

      my_str `str`, optional

        String to add or replace. Required with `ins_after`, `ins_before`, `replace_str`.

      after_flag `str`, optional

        Only replace string in words immediately following this word. Only supported in `arguments` column of `code` table. Default: replace in all.

      ins_after `str`, optional

        Value to insert `my_str` after. Conflicts with `ins_before`, `replace_str`, `combine`.

      ins_before `str`, optional

        Value to insert `my_str` before. Conflicts with `ins_after`, `replace_str`, `combine`.

      replace_str `str`, optional

        Value to replace with `my_str`. Conflicts with `ins_after`, `ins_before`, `combine`.

      combine `bool`, default False

        If true, combine all instances of words after the word in `after_flag`. Conflicts with `ins_after`, `ins_before`, `replace_str`.

    Raises
      ValueError

        for any invalid combination of arguments.

      RuntimeError

        if multiple rows match `my_id`.

    Examples

    All examples assume an open `DButils` instance in `dbu` and an existing code of ID 1. These examples use command line flags but the treatment of strings is general.

        >>> #Replace a string after a flag
        >>> code = dbu.getEntry('Code', 1)
        >>> code.arguments = '-i foobar -j foobar -k foobar'
        >>> dbu.editTable('code', 1, 'arguments', my_str='baz',
        ...               replace_str='bar', after_flag='-j')
        >>> code.arguments
        -i foobar -j foobaz -k foobar

        >>> #Combine multiple instances of a flag into one
        >>> code = dbu.getEntry('Code', 1)
        >>> code.arguments = '-i foo -i bar -j baz'
        >>> dbu.editTable('code', 1, 'arguments', after_flag='-i',
        ...               combine=True)
        >>> code.arguments
        -i foo,bar -j baz

        >>> #Append a string to every instance
        >>> code = dbu.getEntry('Code', 1)
        >>> code.relative_path = 'scripts'
        >>> dbu.editTable('code', 1, 'relative_path', ins_after='scripts',
        ...               my_str='2.0')
        >>> code.relative_path
        scripts2.0

  fileIsNewest(filename, debug=False)

    quesry the database, is this filename or file_id newest version?

    Parameters
      filename `int` or `str`

        filename or file_id

    Returns
      `bool`

        True is file is lastest_version, False is not

  file_id_Clean(invals)

    Given a list of file IDs return only newest versions of matching files.

    Matching is defined as same `product_id` and same `utc_file_date`.

    Parameters
      invals `list` of `int` or of `str`

        All `file_id` or `filename` to check.

    Returns
      :class:`list` of class:int

        Those `file_id` from `invals` which are the newest version of that file.

  getActiveInspectors()

    Query the db and returns all active inspectors

    Returns
      :class:`list` of class:tuple

        For each active inspector, returns full filename (from `relative_path` and `filename`), `description`, `arguments`, and `product`.

  getAllCodes(active=True)

    Return a list of all codes

    Parameters
      active `bool`, default False

        Only return codes which are marked `active_code` and `newest_version`.

    Returns
      `list`

        All codes

  getAllCodesFromProcess(proc_id)

    Given a process id return the code ids that performs that process

    Also returns the valid dates for each code

    Parameters
      proc_id `int`

        `process_id` of process to look up.

    Returns
      :class:`list` of class:tuple

        For every active, newest version code that implements the process, `code_id`, `code_start_date`, and `code_stop_date`.

  getAllFileIds(fullPath=True, startDate=None, endDate=None, level=None, product=None, code=None, instrument=None, exists=None, newest_version=False, limit=None)

    Return all the file ids in the database All parameters are optional; if not specified, default is “all”.

    Parameters
      startDate `datetime`, optional

        First date to include, based on `utc_file_date`

      endDate `datetime`, optional

        Last date to include (inclusive)

      level `float`, optional

        Only include files of this level.

      product `int`, optional

        `product_id` of files to include

      code `int`, optional

        Only return files created by code with ID of `code_id`

      instrument `int`, optional

        Only return files with instrument `instrument_id`

      exists `bool`, default False

        Only return files that exist on disk, based on `exists_on_disk`.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date)

      limit `int`

        Limit number of results, default all

    Returns
      :class:`list` of class:int:

        File ID of all files matching requirements.

    Other Parameters
      fullPath `bool`, default True

        unused

  getAllFilenames(fullPath=True, startDate=None, endDate=None, level=None, product=None, code=None, instrument=None, exists=None, newest_version=False, limit=None)

    Return all the file names in the database

    All parameters are optional; if not specified, default is “all”.

    Parameters
      fullPath `bool`, default True

        Return full path (if False, just filename)

      startDate `datetime`, optional

        First date to include, based on `utc_file_date`

      endDate `datetime`, optional

        Last date to include (inclusive)

      level `float`, optional

        Only include files of this level.

      product `int`, optional

        `product_id` of files to include

      code `int`, optional

        Only return files created by code with ID of `code_id`

      instrument `int`, optional

        Only return files with instrument `instrument_id`

      exists `bool`, default False

        Only return files that exist on disk, based on `exists_on_disk`.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date)

      limit `int`

        Limit number of results, default all

    Returns
      :class:`list` of class:str

        Filename of all files matching requirements.

  getAllInstruments()

    Return dictionaries of instrument traceback dictionaries

    Returns
      `dict`

        dictionaries of instrument traceback dictionaries

  getAllProcesses(timebase='all')

    Get all processes

    Parameters
      timebase `str`, optional

        Limit to products with this `output_timebase` (default: all).

    Returns
      `Query`

        `process` table records

  getAllProducts(id_only=False)

    Return a list of all products as instances

    Parameters
      id_only `bool`, default False

        Return only the `product_id`, instead of the entire record.

    Returns
      :class:`~sqlalchemy.orm.Query` or class:list of `int`

        Complete `product` records for all products, or just `product_id` (if `id_only`).

  getAllSatellites()

    Return dictionaries of satellite, mission objects

    Returns
      `dict`

        dictionaries of satellite, mission objects

  getChildTree(inprod)

    Given an input product return a list of its output product ids

    Parameters
      inprod `int`

        `product_id` of the input product.

    Returns
      :class:`list` of class:int

        `product_id` of all products that can be made from `inprod`.

  getChildrenProcesses(file_id)

    Given a file, return all the processes that use this as input

    Parameters
      file_id `int` or `str`

        `file_id` or `filename`

    Returns
      :class:`list` of class:int

        `process_id` for all processes which can use the given file as input.

  getCodeDirectory()

    Return the code directory for the current mission

    Returns
      `str`

        Code directory for current mission (i.e. `codedir`, if defined).

    See also

    `getDirectory()`
  getCodeFromProcess(proc_id, utc_file_date)

    Given a process id return the code id that performs that process on a particular date.

    Parameters
      proc_id `int`

        `process_id` of process to look up.

      utc_file_date `datetime`

        Date on which the code must be valid.

    Returns
      `int`

        `code_id` for the active, newest version code that implements the process, and is valid on the given date. Returns `None` if there is no match.

    Raises
      DBError

        If there is more than one matching code.

  getCodeID(codename)

    Return the codeID for a code’s filename.

    Parameters
      codename `str` or `int`

        `filename` or `code_id` of code to look up.

    Returns
      `int`

        `code_id` of given code.

  getCodePath(code_id)

    Given a code_id list return the full name (path and all) of the code

    Parameters
      code_id `int` or `str`

        `code_id` or `code_description` of code to look up.

    Returns
      `str`

        Full path to code.

  getCodeVersion(code_id)

    Given a code_id get the code version Given a code_id list return the full name (path and all) of the code

    Parameters
      code_id `int` or `str`

        `code_id` or `code_description` of code to look up.

    Returns
      `Version`

        Version of the code.

  getDirectory(column, default=None)

    Look up directory for the specified column.

    The mission rootdir may be absolute or relative to current path. Directory requested may be in db as absolute or relative to mission root. Home dir references are expanded.

    Parameters
      column `str`

        Name of column in `mission` to look up.

      default `str`, optional

        Default to return if directory not found in mission table, default `None`.

  getEntry(table, args)

    Return entry instance from any table in DB

    Parameters
      table `str`

        Name of the table

      args `int` or `str`

        Argument to identify entry. This is first tried as a primary key (integer or sequence of integers); if that fails, then assumed to be a name and used for a lookup via get[table]ID.

    Returns
      various types

        Matching column from the table. If there is no primary key match and the table does not support name lookup, returns `None`.

    Raises
      DBNoData

        if argument is not found as primary key and name lookup fails (but not if name lookup is not available).

  getErrorPath()

    Return the error directory for the current mission

    Returns
      `str`

        Error directory for current mission (i.e. `errordir`, if defined).

    See also

    `getDirectory()`
  getFileDates(file_id)

    Given a file_id or name return the dates it spans

    Parameters
      file_id `int` or `str`

        `file_id` or `filename` of file to look up.

    Returns
      :class:`list` of class:~datetime.datetime

        First and last UTC timestamp of file.

  getFileFullPath(filename)

    Return the full path to a file given the name or id

    TODO, this is really slow, this query made it a lot faster but I bet it can get better

    Parameters
      filename `str` or `int`

        `filename` or `file_id` of file to look up.

    Returns
      `str`

        Full path to the file.

  getFileID(filename)

    Return the fileID for the input filename

    Parameters
      filename `str` or `int`

        `filename` or `file_id` of file to look up.

    Returns
      `int`

        `file_id` of input file.

  getFileParents(file_id, id_only=False)

    Given a file_id (or filename) return the files that went into making it

    Parameters
      file_id `int` or `str`

        `file_id` or `filename` of the file of interest.

      id_only `bool`, default False

        Return only the `file_id`, instead of the entire record.

    Returns
      :class:`~sqlalchemy.orm.Query` or class:list of `int`

        Complete `file` records for all input files, or just `file_id` (if `id_only`).

  getFileVersion(fileid)

    Return the version instance for a file

    Parameters
      fileid `int` or `str`

        `file_id` or `filename` of the file of interest.

    Returns
      `Version`

        Version of the file.

  getFilecodelink_bycode(code_id)

    Given a code_id return the file_id of all files it created

    Parameters
      code_id `int` or `str`

        `code_id` or `code_description` of the code to look up.

    Returns
      `Query`

        `file_id` of all files created by the code.

  getFilecodelink_byfile(file_id)

    Given a file_id return the code_id that created it, or None

    Parameters
      file_id `int` or `str`

        `file_id` or `filename` of the file to look up.

    Returns
      `int`

        `code_id` of the code that created the file.

  getFiles(startDate=None, endDate=None, level=None, product=None, code=None, instrument=None, exists=None, newest_version=False, limit=None, startTime=None, endTime=None)

    Query database for file records, with filters.

    All parameters are optional; if not specified, default is “all”.

    Parameters
      startDate `datetime`, optional

        First date to include, based on `utc_file_date`

      endDate `datetime`, optional

        Last date to include (inclusive)

      level `float`, optional

        Only include files of this level.

      product `int`, optional

        `product_id` of files to include

      code `int`, optional

        Only return files created by code with ID of `code_id`

      instrument `int`, optional

        Only return files with instrument `instrument_id`

      exists `bool`, default False

        Only return files that exist on disk, based on `exists_on_disk`.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date)

      limit `int`

        Limit number of results, default all

      startTime `datetime`, optional

        Include files containing timestamps at or after this time, `utc_start_time`

      endTime `datetime`, optional

        Include files containing timestamps at or before this time, `utc_stop_time`

    Returns
      `list`

        File records of all files matching requirements.

  getFilesByCode(code_id, newest_version=False, id_only=False)

    Given a code_id (or name) return the files that were created using it

    Parameters
      code_id `int` or `str`

        Only return files created by code with this `code_id` or `code_description`

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date)

      id_only `bool`, default False

        Only return file IDs, not complete file record.

    Returns
      `list`

        File records of all files matching requirements.

  getFilesByDate(daterange, newest_version=False)

    Return files in the db with utc_file_date in the range specified

    Parameters
      daterange `list` of `datetime`

        First and last date to include, based on `utc_file_date`.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date).

    Returns
      `list`

        File records of all files matching requirements.

  getFilesByInstrument(inst_id, level=None, newest_version=False, id_only=False)

    Given an instrument_id return all the file instances associated with it

    Parameters
      inst_id `int` or `str`

        Only return files with this `instrument_id` or `instrument_name`

      level `float`, optional

        Only include files of this level, default all.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date)

      id_only `bool`, default False

        Only return file IDs, not complete file record.

    Returns
      `list`

        File records of all files matching requirements.

  getFilesByProduct(prod_id, newest_version=False)

    Given a product_id or name return all the files associated with it

    Parameters
      prod_id `int` or `str`

        `product_id` or `product_name` of files to include.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date).

    Returns
      `list`

        File records of all files matching requirements.

  getFilesByProductDate(product_id, daterange, newest_version=False)

    Return the files by product id with utc_file_date in range specified

    Parameters
      product_id `int`

        `product_id` of files to include.

      daterange `list` of `datetime`

        First and last date to include, based on `utc_file_date`.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date).

    Returns
      `list`

        File records of all files matching requirements.

  getFilesByProductTime(product_id, daterange, newest_version=False)

    Return the files in the db by product id with any data in date range

    A file with a UTC time range overlapping at all with `daterange` is considered a match, so a returned file may also include some times outside of the range.

    Parameters
      product_id `int`

        `product_id` of files to include.

      daterange `list` of `datetime`

        Range of times to include, based on `utc_start_time` and `utc_stop_time`.

      newest_version `bool`, default False

        Only return files that are the newest version (of their product and date).

    Returns
      `list`

        File records of all files matching requirements.

  getIncomingPath()

    Return the incoming directory for the current mission

    Returns
      `str`

        Incoming directory for current mission (i.e. `incoming_dir`).

    See also

    `getDirectory()`
  getInputProductID(process_id, range=False)

    Return the input products for a particular process.

    Parameters
      process_id `int`

        `process_id` of process to look up.

      range `bool`, default False

        Also return number of days in past/future to use as inputs.

    Returns
      `list`

        Result of query: each element has `input_product_id` and `optional`; if `range`, then also `yesterday` and `tomorrow`.

  getInspectorDirectory()

    Return the inspector directory for the current mission

    Returns
      `str`

        Inspector directory for current mission (i.e. `inspectordir`, if defined).

    See also

    `getDirectory()`
  getInstrumentID(name, satellite_id=None)

    Return the instrument_id for a given instrument.

    Parameters
      name `str` or `int`

        `instrument_name` or `instrument_id`.

      satellite_id `int` or `str`

        Only return results for satellite with this `satellite_id` or `satellite_name`.

    Returns
      `int`

        `instrument_id`.

  getMissionDirectory()

    Return the root directory for the current mission

    Returns
      `str`

        Root directory for current mission (i.e. `rootdir`).

  getMissionID(mission_name)

    Given a mission name return its ID

    Parameters
      mission_name `str`

        Name of mission, i.e. `mission_name`.

    Returns
      `int`

        `mission_id` for the corresponding mission.

    See also

    Missions
  getMissions()

    Return a list of all the missions

    Returns
      :class:`list` of class:str

        Names of all missions in the database.

    Notes

    Ordinarily there is only one mission per database.

  getProcessFromInputProduct(product)

    Given a product id return all the processes that use that as an input

    Use `getProductID()` if have a name (or not sure).

    Parameters
      product `int`

        `product_id` of product.

    Returns
      :class:`list` of class:int

        `process_id` of all processes which use `product` as an input.

  getProcessFromOutputProduct(outProd)

    Gets process from the db that have the output product

    Parameters
      outProd `int`

        `product_id` of product.

    Returns
      `int`

        `process_id` of process which produces `product` as an output.

    Notes

    Assumes there is only one product that makes a process; this is common but not necessarily enforced.

  getProcessID(proc_name)

    Given a process name return its id

    Parameters
      proc_name `str` or `int`

        `process_name` or `process_id`.

    Returns
      `int`

        `process_id`.

  getProcessTimebase(process_id)

    Return the timebase for a process

    Parameters
      process_id `int` or `str`

        `process_id` or `process_name` of the desired process.

    Returns
      `str`

        `output_timebase` for the process.

  getProductID(product_name)

    Return the product ID for an input product name

    Parameters
      product_name `str`

        the name of the product to get the id of. Also supports a sequence of names, a single product ID (to confirm existence), or a sequence of product IDs.

    Returns
      product_id `int`

        the product ID for the input product name

  getProductParentTree()

    go through the db and return a tree of all products and their parents

    This will allow for a run all the non done files script

    Returns
      `list`

        Each entry has two elements: a `product_id` and another list of `product_id` for all product that can be made from it.

    See also

    `getChildTree()`
  getProductsByInstrument(inst_id)

    Get all the products for a Given instrument

    Parameters
      inst_id `int` or `str`

        `instrument_id` or `instrument_name` for instrument

    Returns
      :class:`list` of class:int

        `product_id` for every product associated with this instrument.

  getProductsByLevel(level)

    Get all the products for a Given level

    Parameters
      level `float`

        Data level to look up

    Returns
      :class:`list` of class:int

        `product_id` for every product with `level` equal to `level`.

  getRunProcess()

    Return a list of the processes who’s output_timebase is “RUN”

    Returns
      `list`

        Full `process` record for all `RUN` timebase processes.

  getSatelliteID(sat_name)

    Returns the satellite ID for an input satellite name

    Parameters
      sat_name `str`

        the `satellite_name` to get the id of. Also supports a sequence of names, a single satellite ID (to confirm existence), or a sequence of satellite IDs.

    Returns
      satellite_id `int`

        the `satellite_id` for the input satellite name

  getSatelliteMission(sat_name)

    Given a satellite or satellite id return the mission

    Parameters
      sat_name `int` or `str`

        `satellite_id` or `satellite_name`.

    Returns
      various

        Complete record from `mission` table.

  getTraceback(table, in_id, in_id2=None)

    Master routine for all the getXXXTraceback functions

    The “traceback” is the set of records across tables that are relevant to one particular record

    this is some large select statements with joins in them, these are tested and do work

    Parameters
      table `str`

        Name of the table to look up.

      in_id `int`

        ID, usually primary key on the table, for the record to look up.

    Returns
      `dict`

        Keyed by table name, values are records from that table (instances of table types created by `Table`).

    Other Parameters
      in_id2

        Not used.

    Examples

        >>> tb = dbu.getTraceback('File', 500)
        >>> tb.['product'].product_name
        u'rbspb_int_ect-mageisM35-ns-L05'

  list_release(rel_num, fullpath=True)

    Given a release number return all the filenames with the release

    Parameters
      rel_num `int`

        Release number to list

      fullpath `bool`, default True

        Include full path to files (not just filenames)

    Returns
      :class:`list` of class:str

        All filenames in the release.

  openDB(engine, db_var=None, verbose=False, echo=False)

    Setup python to talk to the database

    Parameters
      engine `str`

        DB engine to connect to

      verbose `bool`, default False

        if True, will print out extra debugging

      echo `bool`, default False

        if True, the Engine will log all statements as well as a repr() of their parameter lists to the logger

    Other Parameters
      db_var

        Does nothing

  purgeProcess(proc, commit=True)

    Remove process and productprocesslink

    Removes a `process` record from the database and all `productprocesslink` records for that process.

    Parameters
      proc `int`

        `process_id` of process to delete.

      commit `bool`, default True

        Commit changes to the database when done.

  renameFile(filename, newname)

    Rename a file in the db

    Parameters
      filename `str` or `int`

        `filename` or `file_id` of file to rename.

      newname `str`

        New name to write to database.

    Notes

    Does not rename the file on disk. Operates on filename only (not entire path).

  resetProcessingFlag(comment)

    Query the db and reset a processing flag

    Parameters
      comment `str`

        the comment to enter into the processing log DB

    Returns
      `bool`

        True - Success, False - Failure

  startLogging()

    Add an entry to the logging table in the DB, logging

  stopLogging(comment)

    Finish the entry to the processing table in the DB, logging

    Parameters
      comment `str`

        a comment to insert into the DB

  tag_release(rel_num)

    Tag all the newest versions of files to a release number (integer)

    Parameters
      rel_num `int`

        Tag all “newest version” files as part of this release.

    See also

    Releases
  updateCodeNewestVersion(code_id, is_newest=False)

    Update a code to indicate whether it’s the newest version.

    Assumption is that the newest version of a code should be the only active one, so sets both `newest_version` and `active_code` fields in the database.

    Parameters
      code_id `int` or `str`

        `code_id` or `code_description` for code to update.

      is_newest `bool`, default False

        Set `newest_version` and `active_code` (True), or not newest, and inactive (False).

  updateInspectorSubs(insp_id)

    Update an existing inspector performing the {} replacements

    Updates the database, replacing the generic `{}` references with the actual values for the inspector.

    Parameters
      insp_id `int`

        `inspector_id` of inspector to update.

  updateProcessSubs(proc_id)

    Update an existing product performing the {} replacements

    Updates the database, replacing the generic `{}` references with the actual values for the process.

    Parameters
      proc_id `int` or `str`

        `process_id` or `process_name` of process to update

  updateProductSubs(product_id)

    Update an existing product performing the {} replacements

    Updates the database, replacing the generic `{}` references with the actual values for the product.

    Parameters
      product_id `int` or `str`

        `product_id` or `product_name` of product to update

Release: 0.1.0 Doc generation date: Feb 10, 2022