LegiScan API Client Downloads

File
Size
LegiScan API Client README

LegiScan API Client

About this Documentation

This is the installation documentation for installing the LegiScan API Client on Linux. This guide is written for installation from source on a Debian platform but any Linux variant should work with minor modifications. There is nothing intrinsically Linux specific to the code, so Windows based servers with access to PHP should work with some modifications, though not tested with MSSQL.

Tested On

Requirements

Further Reading

API Client source documentation is available at: LegiScan API Client Documentation

Additional documentation and data dictionary is available at: LegiScan API Manual

For information about support options and upgrading to Push API please email us at info@legiscan.com, or call 800-618-2750 x401

Files Included

The following files are included in the LegiScan API archive:

Installation

In this example we will install the package under /opt, this could also be your root web server folder such ash /var/www.

Unpack the archive that you downloaded from legiscan.com. This will create a new directory legiscan with the interface that has all the files necessary to interface with the LegiScan API.

cd /opt
wget https://api.legiscan.com/dl/legiscan-current.tar.gz
tar zxvf legiscan-current.tar.gz
cd legiscan

You will need to have a valid API key, if you don't already have one you can get one for free at LegiScan API Registration

Database MySQL

Create the legiscan_api database, add a user legiscan_api with password <your password>, grant rights on the database to the new user and reload privleges.

mysql -u root -p
CREATE DATABASE legiscan_api DEFAULT CHARACTER SET utf8;
GRANT USAGE ON *.* TO 'legiscan_api'@'localhost' IDENTIFIED BY '<your password>';
GRANT CREATE, ALTER, SELECT, INSERT, UPDATE, DELETE ON `legiscan_api`.* TO 'legiscan_api'@'localhost';
FLUSH PRIVILEGES;
exit

Next load the schema and static lookup tables.

mysql -u root -p < schema-mysql.sql

Database PostgreSQL

Create the legiscan_api user with password <your password> since in PostgreSQL the assumption is a database will exist with the same name as the role being used to login.

sudo -u postgres createuser -S -D -R -P legiscan_api
Enter password for new role: <your password>
Enter it again: <your password>

Next create the actual database legiscan_api owned by legiscan_api.

sudo -u postgres createdb -E UTF8 -O legiscan_api legiscan_api

Finally load the schema and static lookup tables.

sudo -u postgres psql -U legiscan_api -f schema-pgsql.sql

Database Diagram

See: Entity Relationship Diagram

Configuration

Using your favorite text editor, open the config.php file and follow the instructions included in the file for setting up configuration options.

vim config.php

Of particular note enter your api_key and change dsn, db_pass to the value set in the above step.

If setting up for pull, update_type and the states list and/or searches lists should be populated. You may also want to tune the global relevance cutoff for searches.

Local document storage is disabled by default, change want_bill_text, want_amendment, want_supplement if needed.

Directories and Permissions

In general the LegiScan Client expects to be ran under a single non-privledged account, often www-data or similar account when running from a web server, or perhaps a custom user/group. Whatever the case, mixing users later on will result in permission conflicts within cache.

There are several directories specified in config.php that need to be writable by the user running the scripts. By default these are located within the legiscan directory, if you select alternate locations please ensure proper permissions and access.

chown -R www-data cache/api cache/doc log signal

NOTE: Be consistent when running from command line e.g., sudo -u www-data php legiscan-cli.php.

NOTE: If middleware_signal is not used or is table based, the signal directory can be ignored.

Pull Daemon

The legiscand.php script provides a daemon process that can use four different methods of keeping a local database synchronized via LegiScan Pull API.

Monitor

This mode will use the ls_monitor table to keep a specific bill_id list updated. This list can be managed with legiscan-cli.php, though determining the bill_id is an exercise for the reader and would require additional scripting, though a likely source would be through the search engine.

php legiscan-cli.php --monitor 823882
php legiscan-cli.php --unmonitor 823882

State

This mode will synchronize the entire master list from one or more states. To set the state list edit config.php and add each state to the states[] setting.

For example to track all legislation in California and US Congress:

states[] = CA
states[] = US

NOTE: We HIGHLY recommend pre-loading the current state dataset with legiscan-bulk.php prior to the first run to minimize the on-boarding queries.

Search (National)

This mode will synchronize the results of searches ran against the national database. To specify the searches edit the config.php and add each search to the searches[] setting.

The searches will also be filtered by the global relevance cutoff setting, which can be overridden on a per search basis by prepending a different score and the pipe | character. In addition a state abbreviations can also be prefixed to override either national or state search. When used with a relevance override the state should appear first separated by a comma ,.

Also notice that the entire search string should be quoted, and any internal quotes should be escaped as \".

searches[] = "gender AND bathroom"
searches[] = "\"national popular vote\""
searches[] = "42|hemp OR cannabis OR marijuana"
searches[] = "NY|charter ADJ schools"
searches[] = "CA,60|vaccination AND status:passed"

State Search

This mode combines both of the other methods such that the searches[] are only ran against the states[] list, unless a search specific state is used.

Bulk Import

Using either the weekly Public Datasets or a custom subscriber bulk-load snapshot, this script will extract the contents of the archive and import/update data as needed.

Process example.zip but do not import, only show what would have been done.

php legiscan-bulk.php --dry-run example.zip

Process and load example.zip

php legiscan-bulk.php example.zip

NOTE: This will not pull local copies of documents unless they are included in the archive, though appropriate stub records will be created.

Push Endpoint

The legiscan-push.php script serves as the endpoint listener for LegiScan Push API services. This will receive payloads from the LegiScan Push server and process them into the provided database. The LegiScan API Client should be installed in a location that is accessible by your web server and externally by LegiScan servers.

For additional authentication you may also set api_auth_key which can be found / updated from the LegiScan API control panel at legiscan.com. This will validate against the HTTP Authorization header sent with incoming payloads and reject those with invalid tokens.

And if your API key is set to receive cooked application/x-www-form-urlencoded payloads, be sure to change the push_form_var setting to the form variable name specified in the API control panel.

NOTE: To utilize this script you will need a Push API subscription.

Command Line Interface

The CLI interface allow for importing specific objects from the command line along with manipulating some internal controls. This could be used for testing purposes or for automation with other custom scripting.

Show the command help

php legiscan-cli.php --help

Get the master list for the most recent session in California, but do not actually import and dump the response payload

php legiscan-cli.php --dry-run --debug --masterlist CA

Request a specific bill id

php legiscan-cli.php --bill 823201

Run a national search and import new bills with a relevance score of 75% or higher

php legiscan-cli.php --search "citizens ADJ united" --state ALL --import new --score 75

Remove a bill_id from the monitor list

php legiscan-cli.php --unmonitor 843038

Add several bill_id to the ignore list

php legiscan-cli.php --ignore 898381,908829,817390

Clean the API cache of stale entries

php legiscan-cli.php --clean --verbose

Web Interface

The web interface is primarily for demonstration purposes to generate single API requests to examine payload structure. The LegiScan API client and legiscan.php should be accessible by your web server. Then point your brower to the URL of interface script (e.g., http://example.com/legiscan/legiscan.php).

Enter your API key in the space provided, select the type of payload request you would like and enter the appropriate ID, then click Run Test.

The system will then output the decoded payload response.

Middleware Signaling

The LegiScan API Client supports signaling a third party application of bill and related data changes so that additional processing can be done. The use case for this would be when the legiscan_api database is used as staging for another custom production system already in place.

This is controlled by the middleware_signal setting and can be either table or directory based.

Table Signaling

When a bill first appears or later updated, or a new text, amendment, supplement, roll call or person record is created a record will be made in the ls_signal table within the legiscan_api database.

This record will contain the object_type, its internal object_id and a processed flag.

The third party application should scan the table for processed = 0, take any necessary actions, then update the record with processed = 1.

Directory Signaling

When a bill first appears or later updated, or a new text, amendment, supplement, roll call or person record is created a file will be created in the signal directory.

This file will be named in the form of object_type.object_id and will contain a JSON object containing the same field along with the updated time stamp.

The third party application should scan the directory for files, take any necessary actions, then delete the file from the signal directory.

Configure init.d and cron job

To install the LegiScan Pull Daemon so that it runs on startup and can be controlled by service. This file presumes the install location is /opt/legiscan, if this is not the case edit the legiscan.service file and change LEGISCAN_PATH.

cp legiscan.service /etc/init.d/legiscan
chmod +x /etc/init.d/legiscan
update-rc.d legiscan defaults
service legiscan start

If running as via the Pull interface, the API cache directory will grow over time. This will create an hourly cron job to clean stale entries.

tee /etc/cron.hourly/legiscan-clean <<EOF
#!/bin/sh
php -q /opt/legiscan/legiscan-cli.php --clean
EOF
chmod +x /etc/cron.hourly/legiscan-clean