Tricoteuses-Senat

Handle French Sénat's open data

Retrieval of open data (in JSON format) from Sénat's website

npx babel-node --extensions ".ts" -- src/scripts/retrieve_open_data.ts --fetch ../senat-data/

Reorganizating open data files and directories into cleaner (and split) directories

npx babel-node --extensions ".ts" -- src/scripts/reorganize_data.ts ../senat-data/

Note: These split files are also available in Tricoteuses / Open data Sénat repositories. They are updated every day.

Validation & cleaning of JSON data

Validation & cleaning of reorganized files

npx babel-node --extensions ".ts" -- src/scripts/clean_reorganized_data.ts ../senat-data/

Note: These split & cleaned files are also available in Tricoteuses / Open data Sénat repositories with the _nettoye suffix. They are updated every day.

Validation & cleaning of big non-split files

npx babel-node --extensions ".ts" --max-old-space-size=8192 -- src/scripts/clean_data.ts ../senat-data/

Note: The big non-split open data files should not be used. Use small split files instead.

Retrieval of députés' pictures from Sénat's website

npx babel-node --extensions ".ts" -- src/scripts/retrieve_deputes_photos.ts --fetch ../senat-data/

Retrieval of députés' non open data informations from Sénat's website

npx babel-node --extensions ".ts" -- src/scripts/retrieve_deputes_infos.ts --fetch --parse ../senat-data/

Retrieval of documents from Sénat's website

npx babel-node --extensions ".ts" -- src/scripts/retrieve_documents.ts --fetch --parse ../senat-data/

Test loading everything in memory

Test loading small split files

npx babel-node --extensions ".ts" --max-old-space-size=2048 -- src/scripts/test_load_all.ts ../senat-data/

Test loading big non-split files

npx babel-node --extensions ".ts" --max-old-space-size=2048 -- src/scripts/test_load_all_big_files.ts ../senat-data/

Note: The big non-split open data files should not be used. Use small split files instead.

Initial generation of TypeScript files from JSON data.

npx quicktype --acronym-style=camel -o src/raw_types/acteurs_et_organes.ts ../senat-data/AMO{10,20,30,40,50}_*.json

npx quicktype --acronym-style=camel -o src/raw_types/agendas.ts ../senat-data/Agenda_{XIV,XV}.json

npx babel-node --extensions ".ts" --max-old-space-size=4096 -- src/scripts/merge_amendements.ts -v ../senat-data/
NODE_OPTIONS=--max-old-space-size=4096 npx quicktype --acronym-style=camel -o src/raw_types/amendements.ts ../senat-data/Amendements_XV_fusionne.json

Edit src/raw_types/amendements.ts to:

• Replace r("ActeurRefElement") with "".
• Remove 2 definitions of ActeurRefElement and replace it with string elsewhere.
• Replace r("GroupePolitiqueRefEnum") with "".
• Remove 2 definitions of GroupePolitiqueRefEnum and replace it with string elsewhere.
• Replace r("Organe") with "".
• Remove 2 definitions of Organe and replace it with string elsewhere.
• Replace r("Libelle") with "".
• Remove 2 definitions of Libelle and replace it with string elsewhere.
• Add:

export interface AmendementWrapper { amendement: Amendement }

* Add:
```typescript
  "AmendementWrapper": o([
      { json: "amendement", js: "amendement", typ: r("Amendement") },
  ], false),

• Add the following static methods to class Convert:

    public static toAmendementWrapper(json: string): AmendementWrapper {
        return cast(JSON.parse(json), r("AmendementWrapper"));
    }
  
    public static amendementWrapperToJson(value: AmendementWrapper): string {
        return JSON.stringify(uncast(value, r("AmendementWrapper")), null, 2);
    }

npx quicktype --acronym-style=camel -o src/raw_types/dossiers_legislatifs.ts ../senat-data/Dossiers_Legislatifs_{XIV,XV}.json

Edit src/raw_types/dossiers_legislatifs.ts to:

• Replace regular expression r\(".*OrganeRef"\) with "".
• Remove definitions of regular expression [^ ]*OrganeRef and replace it with string.
• Replace regular expression r\(".*DossierRef"\) with "".
• Remove definitions of regular expression [^ ]*DossierRef and replace it with string.
• Replace regular expression r\(".*AuteurMotion"\) with "".
• Remove definitions of regular expression [^ ]*AuteurMotion and replace it with string.
• Replace r("DivisionDenominationStructurelle") with "".
• Remove 2 definitions of DivisionDenominationStructurelle and replace it with string elsewhere.

npx babel-node --extensions ".ts" -- src/scripts/merge_scrutins.ts -v ../senat-data/
npx quicktype --acronym-style=camel -o src/raw_types/scrutins.ts ../senat-data/Scrutins_{XIV,XV_fusionne}.json

Edit src/raw_types/scrutins.ts to:

• Replace r("ActeurRef") with "".
• Remove 2 definitions of ActeurRef and replace it with string elsewhere.
• Replace r("GroupeOrganeRef") with "".
• Remove 2 definitions of GroupeOrganeRef and replace it with string elsewhere.
• Replace r("MandatRef") with "".
• Remove 2 definitions of MandatRef and replace it with string elsewhere.
• Replace r("ScrutinOrganeRef") with "".
• Remove 2 definitions of ScrutinOrganeRef and replace it with string elsewhere.
• Replace r("SessionRef") with "".
• Remove 2 definitions of SessionRef and replace it with string elsewhere.
• Add:

export interface ScrutinWrapper { scrutin: Scrutin }

* Add:
```typescript
  "ScrutinWrapper": o([
      { json: "scrutin", js: "scrutin", typ: r("Scrutin") },
  ], false),

• Add the following static methods to class Convert:

    public static toScrutinWrapper(json: string): ScrutinWrapper {
        return cast(JSON.parse(json), r("ScrutinWrapper"));
    }
  
    public static scrutinWrapperToJson(value: ScrutinWrapper): string {
        return JSON.stringify(uncast(value, r("ScrutinWrapper")), null, 2);
    }