eLifePathways /sciencebeam-parser

A set of tools to allow PDF to XML conversion, utilising Apache Beam and other tools. The aim of this project is to bring multiple tools together to generate a full XML document.

★ 299⑂ 32👁 18⑃ 782● 33▤ 21
PythonMITCreated 2017-06-21
Download: v0.1.18v0.1.18 ↓0.1.182026-04-16
+0
Daily
299
stars
32
forks
18
watchers
782
prs
33
issues
21
releases

README

ScienceBeam Parser

 Overview

ScienceBeam Parser allows you to parse scientific documents.
Initially is starting as a partial Python variation of GROBID and allows you to re-use some of the models.
However, it may deviate more in the future.

 Pre-requisites

Docker containers are provided that can be used on multiple operating systems.
It can be used as an example setup for Linux / Ubuntu based systems.

Otherwise the following paragraphs list some of the pre-requisits when not using Docker:

This currently only supports Linux due to the binaries used (pdfalto, wapiti).
It may also be used on other platforms without Docker, provided matching binaries are configured.

For sqlite or lmdb you may need:

bash
apt-get install libsqlite3-dev liblmdb-dev


For Computer Vision PyTorch is required.

For OCR, tesseract needs to be installed. On Ubuntu the following command can be used:

bash
apt-get install tesseract-ocr-eng libtesseract-dev libleptonica-dev


The Word to PDF conversion requires LibreOffice.

 Development

 Create Virtual Environment and install Dependencies

bash
make dev-venv


 Configuration

There is no implicit "grobid-home" directory. The only configuration file is the default config.yml.

Paths may point to local or remote files. Remote files are downloaded and cached locally (urls are assumed to be versioned).

You may override config values using environment variables.
Environment variables should start with SCIENCEBEAM_PARSER__. After that __ is used as a section separator.
For example SCIENCEBEAM_PARSER__LOGGING__HANDLERS__LOG_FILE__LEVEL would override logging.handlers.log_file.level.

Generally, resources and models are loaded on demand, depending on the preload_on_startup configuration option (SCIENCEBEAM_PARSER__PRELOAD_ON_STARTUP environment variable).
Models will be loaded "eagerly" at startup, by setting the configuration option to true.

Which models are used is decided by the profile, and a request may name another one — see Profiles.

No browser page on another origin may call the API as shipped; a deployment names the origins it answers — see Cross-origin requests.

 Sequence models from the Hugging Face Hub

The delft engine takes a model directly from the Hugging Face Hub, as well as from an archive
url. A Hub path is hf://{owner}/{repository}[@{revision}]/{model}, where {model} is a folder
at the root of the repository:

yaml
models:
  header:
    path: 'hf://lfoppiano/grobid-model-header/grobid-header-BidLSTM_CRF_FEATURES-no_embeddings'


The model directory is downloaded to {download_dir}/models/{model}, so it is the existing
download_dir that says where it lands. The revision is optional and, left out, follows the
default branch of the repository: pin one (@v1.1.0, or a commit sha) for a deployment, because
an unpinned model can change underneath it.

{model} is a single path segment, so a repository that groups its models into sub-folders
cannot be addressed this way; such a model is served over https instead

Top issues

Latest release

v0.1.18 v0.1.18 2026-04-16
What's Changed
 Bump cryptography from 46.0.5 to 46.0.6 by @dependabot[bot] in https://github.com/eLifePathways/sciencebeam-parser/pull/579
 Bump pygments from 2.19.2 to 2.20.0 by @dependabot[bot] in https://github.com/eLifePathways/sciencebeam-parser/pull/580
 Bump aiohttp from 3.13.3 to 3.13.4 by @dependabot[bot] in https://github.com/eLifePathways/sciencebeam-parser/pull/581
 Bump cryptography from 46.0.6 to 46.0.7 by @dependabot[bot] in https://github.com/eLifePathways/sciencebeam-parser/pull/582
 Add cache mount for uv in Dockerfile by @HazalCiplak in https://github.com/eLifePathways

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON