Browse Source

Merge branch 'master' into chordpro

Conflicts:
	patacrep/build.py
	patacrep/content/__init__.py
	patacrep/content/song.py
	patacrep/files.py
	patacrep/latex/__init__.py
	patacrep/latex/syntax.py
	patacrep/songs/__init__.py
pull/70/head
Louis 10 years ago
parent
commit
a62cb29216
  1. 2
      MANIFEST.in
  2. 68
      NEWS
  3. 113
      NEWS.md
  4. 9
      patacrep/__init__.py
  5. 2
      patacrep/authors.py
  6. 54
      patacrep/build.py
  7. 6
      patacrep/content/__init__.py
  8. 3
      patacrep/content/cwd.py
  9. 10
      patacrep/content/include.py
  10. 3
      patacrep/content/section.py
  11. 34
      patacrep/content/song.py
  12. 3
      patacrep/content/songsection.py
  13. 3
      patacrep/content/sorted.py
  14. 3
      patacrep/content/tex.py
  15. 20
      patacrep/encoding.py
  16. 2
      patacrep/errors.py
  17. 32
      patacrep/files.py
  18. 8
      patacrep/index.py
  19. 7
      patacrep/latex/__init__.py
  20. 1
      patacrep/latex/detex.py
  21. 22
      patacrep/latex/syntax.py
  22. 154
      patacrep/songbook.py
  23. 84
      patacrep/songs/__init__.py
  24. 13
      patacrep/songs/latex/__init__.py
  25. 18
      patacrep/templates.py
  26. 59
      readme.md
  27. 65
      setup.py
  28. 156
      songbook
  29. 8
      stdeb.cfg

2
MANIFEST.in

@ -0,0 +1,2 @@
include LICENSE NEWS readme.md Requirements.txt
recursive-include patacrep/data *

68
NEWS

@ -1,68 +0,0 @@
songbook 3.7.2
(Louis) Undocumented bug corrections and improvements.
songbook 3.4.7 to 3.7.1
Mainly new songs in the data (which was included in songbook at this
time), and a few undocumented bug corrections and improvements.
songbook (v0.8)
Undocumented.
songbook (v0.7)
(lohrun) New songbook format (not compatible with older version).
Changes have been made to the compilation toolchain that prevent
compilation of old format songbook.
(lohrun) Use LaTeX Songs package v2.10.
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sat, 17 Jul 2010 15:24:14 +0200
songbook (v0.6)
(crep, lohrun) Corrections of mistakes and typos.
(lohrun) Use plain songs package v2.9
(lohrun) Replace makeindex script with a new python version
(lohrun) Add script to produce the list of chords used in songs
(crep, lohrun) Correct chords and gtabs used in songs
(lohrun) Modification of the default geometry
(lohrun) Remove capos from the lyricbook
-- Alexandre Dupas <alexandre.dupas@gmail.com> Fri, 11 Dec 2009 15:35:03 +0100
songbook (0.5)
(crep, lohrun) Corrections of mistakes and typos.
(lohrun) Add a proper volume mechanism
(lohrun) Add volume-1 source containing about 165 songs
(crep) Add naheulbeuk special edition
(lohrun) Upgraded songs.sty with bits from songs package v2.9
(lohrun) Add tabs option
(crep,lohrun) Add lilypond option
-- Alexandre Dupas <alexandre.dupas@gmail.com> Tue, 18 Aug 2009 23:38:12 +0200
songbook (0.4)
(crep, lohrun) Corrections of mistakes and typos.
(crep, lohrun) Add cover picture to each song
(lohrun) Update to the Songs Package v2.8
(lohrun) Update makefile to be POSIX compilant
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sun, 31 May 2009 01:39:16 +0200
songbook (0.3)
(crep) Corrections of a lot of mistakes.
(crep) Include image support.
(lohrun) Add make-html utility.
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sun, 15 Feb 2009 18:34:59 +0100
songbook (0.2)
Initial version.
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sat, 11 Oct 2008 20:00:00 +0100

113
NEWS.md

@ -0,0 +1,113 @@
# patacrep 4.0.0
* Project gestion
* Name change [#39](http://github.com/patacrep/patacrep/issues/39)
* Renew of the developement team
* Separation engine/data
* The engine is the [current poject](http://github.com/patacrep/patacrep)
* Data have their [own project](http://github.com/patacrep/patadata)
* And so does [various tools](http://github.com/patacrep/pataextra)
* Internal changes
* Complete migration to Python
* No more Makefiles
* Creation of a `songbook` command
* patacrep uses Python3 [#65](http://github.com/patacrep/patacrep/issues/65)
* Massive code refactoring and simplification
* [PEP8](http://legacy.python.org/dev/peps/pep-0008/) conformity
* Better LaTeX Packages
* Better langages handling
* Better errors handling
* Better code documentation (in comments)
* Caching song AST, which gave an improvement of 45s for the compilation of all patadata [#51](http://github.com/patacrep/patacrep/issues/51)
* Lot of small improvements
* Installation
* All from PyPi ! You can now use pip to install/update/remove patacrep
* Fonctionnalities
* Change the template engine [#9](http://github.com/patacrep/patacrep/issues/9)
* Ability to add user variables [#18](http://github.com/patacrep/patacrep/issues/18)
* Change the song inclusion syntaxe [#47](http://github.com/patacrep/patacrep/issues/47)
* It is now possible to include other things than songs
* You can write a plugins to include your own type of content
* Personalisaitons of the songbook is easier with patadata templates (font, paper, colors, column, ...) [#41](http://github.com/patacrep/patacrep/issues/41)
* You can change the number of columns [#41](http://github.com/patacrep/patacrep/issues/41)
* Lilypond
* On the fly lylipond files compilation
* Adapt partition size to the paper size [#19](http://github.com/patacrep/patacrep/issues/19)
* You can choos how to sort the songs [#36](http://github.com/patacrep/patacrep/issues/36)
* Easier song repertories management [#43](http://github.com/patacrep/patacrep/issues/43) and [#45](http://github.com/patacrep/patacrep/issues/45)
* You can have more than one data folder
* Better index customisation
* Better gestion of files encoding [#62](http://github.com/patacrep/patacrep/issues/62).
# songbook 3.7.2
(Louis) Undocumented bug corrections and improvements.
# songbook 3.4.7 to 3.7.1
Mainly new songs in the data (which was included in songbook at this
time), and a few undocumented bug corrections and improvements.
# songbook (v0.8)
Undocumented.
# songbook (v0.7)
(lohrun) New songbook format (not compatible with older version).
Changes have been made to the compilation toolchain that prevent
compilation of old format songbook.
(lohrun) Use LaTeX Songs package v2.10.
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sat, 17 Jul 2010 15:24:14 +0200
# songbook (v0.6)
(crep, lohrun) Corrections of mistakes and typos.
(lohrun) Use plain songs package v2.9
(lohrun) Replace makeindex script with a new python version
(lohrun) Add script to produce the list of chords used in songs
(crep, lohrun) Correct chords and gtabs used in songs
(lohrun) Modification of the default geometry
(lohrun) Remove capos from the lyricbook
-- Alexandre Dupas <alexandre.dupas@gmail.com> Fri, 11 Dec 2009 15:35:03 +0100
# songbook (0.5)
(crep, lohrun) Corrections of mistakes and typos.
(lohrun) Add a proper volume mechanism
(lohrun) Add volume-1 source containing about 165 songs
(crep) Add naheulbeuk special edition
(lohrun) Upgraded songs.sty with bits from songs package v2.9
(lohrun) Add tabs option
(crep,lohrun) Add lilypond option
-- Alexandre Dupas <alexandre.dupas@gmail.com> Tue, 18 Aug 2009 23:38:12 +0200
# songbook (0.4)
(crep, lohrun) Corrections of mistakes and typos.
(crep, lohrun) Add cover picture to each song
(lohrun) Update to the Songs Package v2.8
(lohrun) Update makefile to be POSIX compilant
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sun, 31 May 2009 01:39:16 +0200
# songbook (0.3)
(crep) Corrections of a lot of mistakes.
(crep) Include image support.
(lohrun) Add make-html utility.
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sun, 15 Feb 2009 18:34:59 +0100
# songbook (0.2)
Initial version.
-- Alexandre Dupas <alexandre.dupas@gmail.com> Sat, 11 Oct 2008 20:00:00 +0100

9
patacrep/__init__.py

@ -3,9 +3,14 @@
from pkg_resources import resource_filename from pkg_resources import resource_filename
import os import os
# Version # Check Python version
import sys
if sys.version_info < (3, 3):
print("ERROR: Your Python version is too old. Please use a Python version > 3.3.")
sys.exit(1)
__TUPLE_VERSION__ = (4, 0, 0, "alpha") # Patacrep version.
__TUPLE_VERSION__ = (4, 0, 0)
__version__ = '.'.join([str(number) for number in __TUPLE_VERSION__]) __version__ = '.'.join([str(number) for number in __TUPLE_VERSION__])
# Directory containing shared data (default templates, custom LaTeX packages, # Directory containing shared data (default templates, custom LaTeX packages,

2
patacrep/authors.py

@ -1,5 +1,3 @@
# -*- coding: utf-8 -*-
"""Authors string management.""" """Authors string management."""
import logging import logging

54
patacrep/build.py

@ -1,15 +1,14 @@
# -*- coding: utf-8 -*-
"""Build a songbook, according to parameters found in a .sb file.""" """Build a songbook, according to parameters found in a .sb file."""
import codecs import codecs
import copy import copy
import glob import glob
import logging import logging
import threading
import os.path import os.path
from subprocess import Popen, PIPE, call from subprocess import Popen, PIPE, call
from patacrep import __DATADIR__, authors, content, errors from patacrep import __DATADIR__, authors, content, errors, files
from patacrep.index import process_sxd from patacrep.index import process_sxd
from patacrep.templates import TexRenderer from patacrep.templates import TexRenderer
from patacrep.songs import DataSubpath from patacrep.songs import DataSubpath
@ -33,6 +32,7 @@ DEFAULT_CONFIG = {
'lang': 'english', 'lang': 'english',
'content': [], 'content': [],
'titleprefixwords': [], 'titleprefixwords': [],
'encoding': None,
} }
@ -91,6 +91,7 @@ class Songbook(object):
config['template'], config['template'],
config['datadir'], config['datadir'],
config['lang'], config['lang'],
config['encoding'],
) )
config.update(renderer.get_variables()) config.update(renderer.get_variables())
config.update(self.config) config.update(self.config)
@ -99,8 +100,19 @@ class Songbook(object):
copy.deepcopy(config['authwords']) copy.deepcopy(config['authwords'])
) )
# Configuration set # Loading custom plugins
config['_content_plugins'] = files.load_plugins(
datadirs=config.get('datadir', []),
root_modules=['content'],
keyword='CONTENT_PLUGINS',
)
config['_song_plugins'] = files.load_plugins(
datadirs=config.get('datadir', []),
root_modules=['songs'],
keyword='SONG_PARSERS',
)
# Configuration set
config['render_content'] = content.render_content config['render_content'] = content.render_content
config['content'] = content.process_content( config['content'] = content.process_content(
config.get('content', []), config.get('content', []),
@ -110,6 +122,13 @@ class Songbook(object):
renderer.render_tex(output, config) renderer.render_tex(output, config)
def _log_pipe(pipe):
"""Log content from `pipe`."""
while 1:
line = pipe.readline()
if not bool(line):
break
LOGGER.debug(line.strip())
class SongbookBuilder(object): class SongbookBuilder(object):
"""Provide methods to compile a songbook.""" """Provide methods to compile a songbook."""
@ -200,22 +219,33 @@ class SongbookBuilder(object):
stdin=PIPE, stdin=PIPE,
stdout=PIPE, stdout=PIPE,
stderr=PIPE, stderr=PIPE,
env=os.environ,
universal_newlines=True, universal_newlines=True,
) env=os.environ)
except Exception as error: except Exception as error:
LOGGER.debug(error) LOGGER.debug(error)
raise errors.LatexCompilationError(self.basename) raise errors.LatexCompilationError(self.basename)
if not self.interactive: if not self.interactive:
process.stdin.close() process.stdin.close()
log = ''
line = process.stdout.readline()
while line:
log += str(line)
line = process.stdout.readline()
LOGGER.debug(log)
standard_output = threading.Thread(
target=_log_pipe,
kwargs={
'pipe' : process.stdout,
}
)
standard_error = threading.Thread(
target=_log_pipe,
kwargs={
'pipe' : process.stderr,
}
)
standard_output.daemon = True
standard_error.daemon = True
standard_error.start()
standard_output.start()
standard_error.join()
standard_output.join()
process.wait() process.wait()
if process.returncode: if process.returncode:

6
patacrep/content/__init__.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Content plugin management. """Content plugin management.
Content that can be included in a songbook is controlled by plugins. From the Content that can be included in a songbook is controlled by plugins. From the
@ -69,7 +66,6 @@ More documentation in the docstring of Content.
""" """
import glob import glob
import importlib
import jinja2 import jinja2
import logging import logging
import os import os
@ -177,7 +173,7 @@ def process_content(content, config=None):
included in the .tex file. included in the .tex file.
""" """
contentlist = [] contentlist = []
plugins = files.load_plugins(config, ["content"], "CONTENT_PLUGINS") plugins = config.get('_content_plugins', {})
keyword_re = re.compile(r'^ *(?P<keyword>\w*) *(\((?P<argument>.*)\))? *$') keyword_re = re.compile(r'^ *(?P<keyword>\w*) *(\((?P<argument>.*)\))? *$')
if not content: if not content:
content = [["song"]] content = [["song"]]

3
patacrep/content/cwd.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Change base directory before importing songs.""" """Change base directory before importing songs."""
from patacrep.content import process_content from patacrep.content import process_content

10
patacrep/content/include.py

@ -1,5 +1,3 @@
# -*- coding: utf-8 -*-
"""Include an external list of songs """Include an external list of songs
This plugin provides keyword 'include', used to include an external list of This plugin provides keyword 'include', used to include an external list of
@ -48,15 +46,15 @@ def parse(keyword, config, argument, contentlist):
filepath = load_from_datadirs(path, config) filepath = load_from_datadirs(path, config)
content_file = None content_file = None
try: try:
content_file = encoding.open_read(filepath, 'r') with encoding.open_read(
filepath,
encoding=config['encoding']
) as content_file:
new_content = json.load(content_file) new_content = json.load(content_file)
except Exception as error: # pylint: disable=broad-except except Exception as error: # pylint: disable=broad-except
LOGGER.error(error) LOGGER.error(error)
LOGGER.error("Error while loading file '{}'.".format(filepath)) LOGGER.error("Error while loading file '{}'.".format(filepath))
sys.exit(1) sys.exit(1)
finally:
if content_file:
content_file.close()
config["datadir"].append(os.path.abspath(os.path.dirname(filepath))) config["datadir"].append(os.path.abspath(os.path.dirname(filepath)))
new_contentlist += process_content(new_content, config) new_contentlist += process_content(new_content, config)

3
patacrep/content/section.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Allow LaTeX sections (starred or not) as content of a songbook.""" """Allow LaTeX sections (starred or not) as content of a songbook."""
from patacrep.content import Content, ContentError from patacrep.content import Content, ContentError

34
patacrep/content/song.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Plugin to include songs to the songbook.""" """Plugin to include songs to the songbook."""
import glob import glob
@ -8,7 +5,7 @@ import jinja2
import logging import logging
import os import os
from patacrep.content import Content, process_content, ContentError from patacrep.content import process_content, ContentError, Content
from patacrep import files, errors from patacrep import files, errors
LOGGER = logging.getLogger(__name__) LOGGER = logging.getLogger(__name__)
@ -50,20 +47,17 @@ def parse(keyword, argument, contentlist, config):
expressions (interpreted using the glob module), referring to songs. expressions (interpreted using the glob module), referring to songs.
- config: the current songbook configuration dictionary. - config: the current songbook configuration dictionary.
Return a list of SongRenderer() instances. Return a list of Song() instances.
""" """
plugins = files.load_plugins(config, ["songs"], "SONG_PARSERS") plugins = config['_song_plugins']
if '_languages' not in config: if '_languages' not in config:
config['_languages'] = set() config['_languages'] = set()
songlist = [] songlist = []
for songdir in config['_songdir']: for songdir in config['_songdir']:
if contentlist: if contentlist:
break break
contentlist = [ contentlist = files.recursive_find(songdir.fullpath, plugins.keys())
filename
for filename
in files.recursive_find(songdir.fullpath, plugins.keys())
]
for elem in contentlist: for elem in contentlist:
before = len(songlist) before = len(songlist)
for songdir in config['_songdir']: for songdir in config['_songdir']:
@ -71,22 +65,22 @@ def parse(keyword, argument, contentlist, config):
continue continue
with files.chdir(songdir.datadir): with files.chdir(songdir.datadir):
for filename in glob.iglob(os.path.join(songdir.subpath, elem)): for filename in glob.iglob(os.path.join(songdir.subpath, elem)):
LOGGER.debug('Parsing file "{}"'.format(filename))
extension = filename.split(".")[-1] extension = filename.split(".")[-1]
if extension not in plugins: try:
renderer = SongRenderer(plugins[extension](
songdir.datadir,
filename,
config,
))
except KeyError:
LOGGER.warning(( LOGGER.warning((
'File "{}" does not end with one of {}. Ignored.' 'I do not know how to parse "{}". Ignored.'
).format( ).format(
os.path.join(songdir.datadir, filename), os.path.join(songdir.datadir, filename),
", ".join(["'.{}'".format(key) for key in plugins.keys()]),
) )
) )
continue continue
LOGGER.debug('Parsing file "{}"'.format(filename))
renderer = SongRenderer(plugins[extension](
songdir.datadir,
filename,
config,
))
songlist.append(renderer) songlist.append(renderer)
config["_languages"].update(renderer.song.languages) config["_languages"].update(renderer.song.languages)
if len(songlist) > before: if len(songlist) > before:

3
patacrep/content/songsection.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Allow 'songchapter' and 'songsection' as content of a songbook.""" """Allow 'songchapter' and 'songsection' as content of a songbook."""
from patacrep.content import Content, ContentError from patacrep.content import Content, ContentError

3
patacrep/content/sorted.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Sorted list of songs. """Sorted list of songs.
This plugin provides keyword 'sorted', used to include a sorted list of songs This plugin provides keyword 'sorted', used to include a sorted list of songs

3
patacrep/content/tex.py

@ -1,6 +1,3 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Include LaTeX raw code in the songbook.""" """Include LaTeX raw code in the songbook."""
import logging import logging

20
patacrep/encoding.py

@ -1,21 +1,29 @@
# -*- coding: utf-8 -*-
"""Dealing with encoding problems.""" """Dealing with encoding problems."""
import codecs import codecs
import chardet import chardet
import logging import logging
import contextlib
LOGGER = logging.getLogger(__name__) LOGGER = logging.getLogger(__name__)
def open_read(filename, mode='r'):
@contextlib.contextmanager
def open_read(filename, mode='r', encoding=None):
"""Open a file for reading, guessing the right encoding. """Open a file for reading, guessing the right encoding.
Return a fileobject, reading unicode strings. Return a fileobject, reading unicode strings.
If `encoding` is set, use it as the encoding (do not guess).
""" """
return codecs.open( if encoding is None:
fileencoding = chardet.detect(open(filename, 'rb').read())['encoding']
else:
fileencoding = encoding
with codecs.open(
filename, filename,
mode=mode, mode=mode,
encoding=chardet.detect(open(filename, 'rb').read())['encoding'], encoding=fileencoding,
errors='replace', errors='replace',
) ) as fileobject:
yield fileobject

2
patacrep/errors.py

@ -1,5 +1,3 @@
# -*- coding: utf-8 -*-
"""Songbook exceptions and errors.""" """Songbook exceptions and errors."""
class SongbookError(Exception): class SongbookError(Exception):

32
patacrep/files.py

@ -1,31 +1,32 @@
# -*- coding: utf-8 -*-
"""File system utilities.""" """File system utilities."""
from contextlib import contextmanager from contextlib import contextmanager
import fnmatch
import importlib import importlib
import logging import logging
import os import os
import posixpath import posixpath
import re
LOGGER = logging.getLogger(__name__) LOGGER = logging.getLogger(__name__)
def recursive_find(root_directory, patterns): def recursive_find(root_directory, extensions):
"""Recursively find files matching one of the patterns, from a root_directory. """Recursively find files with some extension, from a root_directory.
Return a list of files matching one of the patterns. Return a list of files matching those conditions.
Arguments:
- `extensions`: list of accepted extensions.
- `root_directory`: root directory of the search.
""" """
if not os.path.isdir(root_directory): if not os.path.isdir(root_directory):
return [] return []
matches = [] matches = []
pattern = re.compile(r'.*\.({})$'.format('|'.join(extensions)))
with chdir(root_directory): with chdir(root_directory):
for root, __ignored, filenames in os.walk(os.curdir): for root, __ignored, filenames in os.walk(os.curdir):
for pattern in patterns: for filename in filenames:
for filename in fnmatch.filter( if pattern.match(filename):
filenames,
"*.{}".format(pattern),
):
matches.append(os.path.join(root, filename)) matches.append(os.path.join(root, filename))
return matches return matches
@ -68,11 +69,15 @@ def chdir(path):
else: else:
yield yield
def load_plugins(config, root_modules, keyword): def load_plugins(datadirs, root_modules, keyword):
"""Load all plugins, and return a dictionary of those plugins. """Load all plugins, and return a dictionary of those plugins.
A plugin is a .py file, submodule of `subdir`, located in one of the
directories of `datadirs`. It contains a dictionary `keyword`. The return
value is the union of the dictionaries of the loaded plugins.
Arguments: Arguments:
- config: the configuration dictionary of the songbook - datadirs: List of directories in which plugins are to be searched.
- root_modules: the submodule in which plugins are to be searched, as a - root_modules: the submodule in which plugins are to be searched, as a
list of modules (e.g. ["some", "deep", "module"] for list of modules (e.g. ["some", "deep", "module"] for
"some.deep.module"). "some.deep.module").
@ -87,7 +92,7 @@ def load_plugins(config, root_modules, keyword):
directory_list = ( directory_list = (
[ [
os.path.join(datadir, "python", *root_modules) os.path.join(datadir, "python", *root_modules)
for datadir in config.get('datadir', []) for datadir in datadirs
] ]
+ [os.path.join( + [os.path.join(
os.path.dirname(__file__), os.path.dirname(__file__),
@ -124,4 +129,3 @@ def load_plugins(config, root_modules, keyword):
continue continue
plugins[key] = value plugins[key] = value
return plugins return plugins

8
patacrep/index.py

@ -1,5 +1,3 @@
# -*- coding: utf-8 -*-
"""Manage indexes. """Manage indexes.
Generate indexes files for the songbook compilation. This is a replacement for Generate indexes files for the songbook compilation. This is a replacement for
@ -29,13 +27,9 @@ def process_sxd(filename):
""" """
data = [] data = []
index_file = None index_file = None
try: with encoding.open_read(filename) as index_file:
index_file = encoding.open_read(filename, 'r')
for line in index_file: for line in index_file:
data.append(line.strip()) data.append(line.strip())
finally:
if index_file:
index_file.close()
i = 1 i = 1
idx = Index(data[0]) idx = Index(data[0])

7
patacrep/latex/__init__.py

@ -1,3 +1,8 @@
"""Dumb and very very incomplete LaTeX parser.""" """Dumb and very very incomplete LaTeX parser.
This module uses an LALR parser to try to parse LaTeX code. LaTeX language
*cannot* be parsed by an LALR parser, so this is a very simple attemps, which
will work on simple cases, but not on complex ones.
"""
from patacrep.latex.syntax import tex2plain, parse_song from patacrep.latex.syntax import tex2plain, parse_song

1
patacrep/latex/detex.py

@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
"""Render `very simple` TeX commands in a simple TeX code.""" """Render `very simple` TeX commands in a simple TeX code."""
import logging import logging

22
patacrep/latex/syntax.py

@ -41,7 +41,8 @@ class Parser:
def p_error(self, token): def p_error(self, token):
"""Manage parsing errors.""" """Manage parsing errors."""
LOGGER.error("Erreur fichier {}, ligne {}, position {}.".format( LOGGER.error(
"Error in file {}, line {} at position {}.".format(
str(self.filename), str(self.filename),
token.lineno, token.lineno,
self.__find_column(token), self.__find_column(token),
@ -223,13 +224,20 @@ class Parser:
else: else:
symbols[0] = symbols[2].prepend(symbols[1]) symbols[0] = symbols[2].prepend(symbols[1])
def silent_yacc(*args, **kwargs):
"""Call yacc, suppressing (as far as possible) output and generated files.
"""
return yacc.yacc(
write_tables=0,
debug=0,
*args,
**kwargs
)
def tex2plain(string): def tex2plain(string):
"""Parse string and return its plain text version.""" """Parse string and return its plain text version."""
return detex( return detex(
yacc.yacc( silent_yacc(
write_tables=0,
debug=0,
module=Parser(), module=Parser(),
).parse( ).parse(
string, string,
@ -246,11 +254,7 @@ def parse_song(content, filename=None):
display error messages. display error messages.
""" """
return detex( return detex(
yacc.yacc( silent_yacc(module=Parser(filename)).parse(
module=Parser(filename),
write_tables=0,
debug=0,
).parse(
content, content,
lexer=SongLexer().lexer, lexer=SongLexer().lexer,
).metadata ).metadata

154
patacrep/songbook.py

@ -0,0 +1,154 @@
"""Command line tool to compile songbooks using the songbook library."""
import argparse
import json
import locale
import logging
import os.path
import textwrap
import sys
from patacrep.build import SongbookBuilder, DEFAULT_STEPS
from patacrep import __version__
from patacrep import errors
import patacrep.encoding
# Logging configuration
logging.basicConfig(level=logging.INFO)
LOGGER = logging.getLogger()
# pylint: disable=too-few-public-methods
class ParseStepsAction(argparse.Action):
"""Argparse action to split a string into a list."""
def __call__(self, __parser, namespace, values, __option_string=None):
if not getattr(namespace, self.dest):
setattr(namespace, self.dest, [])
setattr(
namespace,
self.dest,
(
getattr(namespace, self.dest)
+ [value.strip() for value in values[0].split(',')]
),
)
class VerboseAction(argparse.Action):
"""Set verbosity level with option --verbose."""
def __call__(self, *_args, **_kwargs):
LOGGER.setLevel(logging.DEBUG)
def argument_parser(args):
"""Parse arguments"""
parser = argparse.ArgumentParser(description="A song book compiler")
parser.add_argument('--version', help='Show version', action='version',
version='%(prog)s ' + __version__)
parser.add_argument('book', nargs=1, help=textwrap.dedent("""\
Book to compile.
"""))
parser.add_argument('--datadir', '-d', nargs='+', type=str, action='append',
help=textwrap.dedent("""\
Data location. Expected (not necessarily required)
subdirectories are 'songs', 'img', 'latex', 'templates'.
"""))
parser.add_argument('--verbose', '-v', nargs=0, action=VerboseAction,
help=textwrap.dedent("""\
Show details about the compilation process.
"""))
parser.add_argument('--steps', '-s', nargs=1, type=str,
action=ParseStepsAction,
help=textwrap.dedent("""\
Steps to run. Default is "{steps}".
Available steps are:
"tex" produce .tex file from templates;
"pdf" compile .tex file;
"sbx" compile index files;
"clean" remove temporary files;
any string beginning with '%%' (in this case, it will be run
in a shell). Several steps (excepted the custom shell
command) can be combinend in one --steps argument, as a
comma separated string.
""".format(steps=','.join(DEFAULT_STEPS))),
default=None,
)
options = parser.parse_args(args)
return options
def main():
"""Main function:"""
# set script locale to match user's
try:
locale.setlocale(locale.LC_ALL, '')
except locale.Error as error:
# Locale is not installed on user's system, or wrongly configured.
LOGGER.error("Locale error: {}\n".format(str(error)))
options = argument_parser(sys.argv[1:])
songbook_path = options.book[0]
basename = os.path.basename(songbook_path)[:-3]
try:
with patacrep.encoding.open_read(songbook_path) as songbook_file:
songbook = json.load(songbook_file)
if 'encoding' in songbook:
with patacrep.encoding.open_read(
songbook_path,
encoding=songbook['encoding']
) as songbook_file:
songbook = json.load(songbook_file)
except Exception as error: # pylint: disable=broad-except
LOGGER.error(error)
LOGGER.error("Error while loading file '{}'.".format(songbook_path))
sys.exit(1)
# Gathering datadirs
datadirs = []
if options.datadir:
# Command line options
datadirs += [item[0] for item in options.datadir]
if 'datadir' in songbook:
# .sg file
if isinstance(songbook['datadir'], str):
songbook['datadir'] = [songbook['datadir']]
datadirs += [
os.path.join(
os.path.dirname(os.path.abspath(songbook_path)),
path
)
for path in songbook['datadir']
]
# Default value
datadirs.append(os.path.dirname(os.path.abspath(songbook_path)))
songbook['datadir'] = datadirs
try:
sb_builder = SongbookBuilder(songbook, basename)
sb_builder.unsafe = True
sb_builder.build_steps(options.steps)
except errors.SongbookError as error:
LOGGER.error(error)
if LOGGER.level >= logging.INFO:
LOGGER.error(
"Running again with option '-v' may give more information."
)
sys.exit(1)
except KeyboardInterrupt:
LOGGER.warning("Aborted by user.")
sys.exit(1)
sys.exit(0)
if __name__ == '__main__':
main()

84
patacrep/songs/__init__.py

@ -1,9 +1,8 @@
# -*- coding: utf-8 -*-
"""Song management.""" """Song management."""
import errno import errno
import hashlib import hashlib
import jinja2
import logging import logging
import os import os
import pickle import pickle
@ -11,6 +10,7 @@ import re
from patacrep.authors import processauthors from patacrep.authors import processauthors
from patacrep import files, encoding from patacrep import files, encoding
from patacrep.content import Content
LOGGER = logging.getLogger(__name__) LOGGER = logging.getLogger(__name__)
@ -62,21 +62,33 @@ class DataSubpath(object):
self.subpath = os.path.join(self.subpath, path) self.subpath = os.path.join(self.subpath, path)
return self return self
# pylint: disable=too-few-public-methods, too-many-instance-attributes # pylint: disable=too-many-instance-attributes
class Song(object): class Song(Content):
"""Song management""" """Song (or song metadata)
This class represents a song, bound to a file.
- It can parse the file given in arguments.
- It can render the song as some LaTeX code.
- Its content is cached, so that if the file has not been changed, the
file is not parsed again.
This class is inherited by classes implementing song management for
several file formats. Those subclasses must implement:
- `parse()` to parse the file;
- `render()` to render the song as LaTeX code.
"""
# Version format of cached song. Increment this number if we update # Version format of cached song. Increment this number if we update
# information stored in cache. # information stored in cache.
CACHE_VERSION = 0 CACHE_VERSION = 1
# List of attributes to cache # List of attributes to cache
cached_attributes = [ cached_attributes = [
"titles", "titles",
"unprefixed_titles", "unprefixed_titles",
"cached",
"data", "data",
"datadir",
"fullpath",
"subpath", "subpath",
"languages", "languages",
"authors", "authors",
@ -84,14 +96,11 @@ class Song(object):
"_version", "_version",
] ]
# Default data
DEFAULT_DATA = {
'@titles': [],
'@languages': [],
}
def __init__(self, datadir, subpath, config): def __init__(self, datadir, subpath, config):
self.fullpath = os.path.join(datadir, subpath) self.fullpath = os.path.join(datadir, subpath)
self.datadir = datadir
self.encoding = config["encoding"]
if datadir: if datadir:
# Only songs in datadirs are cached # Only songs in datadirs are cached
self._filehash = hashlib.md5( self._filehash = hashlib.md5(
@ -116,14 +125,13 @@ class Song(object):
)) ))
# Data extraction from the latex song # Data extraction from the latex song
self.data = self.DEFAULT_DATA self.titles = []
self.data['@path'] = self.fullpath self.data = {}
self.data.update(self.parse( self.parse()
encoding.open_read(self.fullpath).read()
)) # Post processing of data
self.titles = self.data['@titles']
self.languages = self.data['@languages']
self.datadir = datadir self.datadir = datadir
self.subpath = subpath
self.unprefixed_titles = [ self.unprefixed_titles = [
unprefixed_title( unprefixed_title(
title, title,
@ -132,14 +140,15 @@ class Song(object):
for title for title
in self.titles in self.titles
] ]
self.subpath = subpath
if "by" in self.data:
self.authors = processauthors( self.authors = processauthors(
self.data["by"], self.authors,
**config["_compiled_authwords"] **config["_compiled_authwords"]
) )
else:
self.authors = [] # Cache management
#: Special attribute to allow plugins to store cached data
self.cached = None
self._version = self.CACHE_VERSION self._version = self.CACHE_VERSION
self._write_cache() self._write_cache()
@ -165,11 +174,24 @@ class Song(object):
Arguments: Arguments:
- output: Name of the output file. - output: Name of the output file.
""" """
return NotImplementedError() raise NotImplementedError()
def parse(self, content): # pylint: disable=no-self-use, unused-argument def parse(self): # pylint: disable=no-self-use
"""Parse song, and return a dictionary of its data.""" """Parse song.
return NotImplementedError()
It set the following attributes:
- titles: the list of (raw) titles. This list will be processed to
remove prefixes.
- languages: the list of languages used in the song, as languages
recognized by the LaTeX babel package.
- authors: the list of (raw) authors. This list will be processed to
'clean' it (see function :func:`patacrep.authors.processauthors`).
- data: song metadata. Used (among others) to sort the songs.
- cached: additional data that will be cached. Thus, data stored in
this attribute must be picklable.
"""
raise NotImplementedError()
def unprefixed_title(title, prefixes): def unprefixed_title(title, prefixes):
"""Remove the first prefix of the list in the beginning of title (if any). """Remove the first prefix of the list in the beginning of title (if any).

13
patacrep/songs/latex/__init__.py

@ -7,16 +7,23 @@ will work on simple cases, but not on complex ones.
import os import os
from patacrep import files from patacrep import files, encoding
from patacrep.latex import parse_song from patacrep.latex import parse_song
from patacrep.songs import Song from patacrep.songs import Song
class LatexSong(Song): class LatexSong(Song):
"""LaTeX song parser.""" """LaTeX song parser."""
def parse(self, content): def parse(self):
"""Parse content, and return the dictinory of song data.""" """Parse content, and return the dictinory of song data."""
return parse_song(content, self.fullpath) with encoding.open_read(self.fullpath, encoding=self.encoding) as song:
self.data = parse_song(song.read(), self.fullpath)
self.titles = self.data['@titles']
del self.data['@titles']
self.languages = self.data['@languages']
del self.data['@languages']
self.authors = self.data['by']
del self.data['by']
def tex(self, output): def tex(self, output):
"""Return the LaTeX code rendering the song.""" """Return the LaTeX code rendering the song."""

18
patacrep/templates.py

@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
"""Template for .tex generation settings and utilities""" """Template for .tex generation settings and utilities"""
from jinja2 import Environment, FileSystemLoader, ChoiceLoader, \ from jinja2 import Environment, FileSystemLoader, ChoiceLoader, \
@ -9,7 +8,8 @@ import os
import re import re
import json import json
from patacrep import encoding, errors, files from patacrep import errors, files
import patacrep.encoding
_LATEX_SUBS = ( _LATEX_SUBS = (
(re.compile(r'\\'), r'\\textbackslash'), (re.compile(r'\\'), r'\\textbackslash'),
@ -67,7 +67,7 @@ def _escape_tex(value):
class TexRenderer(object): class TexRenderer(object):
"""Render a template to a LaTeX file.""" """Render a template to a LaTeX file."""
def __init__(self, template, datadirs, lang): def __init__(self, template, datadirs, lang, encoding=None):
'''Start a new jinja2 environment for .tex creation. '''Start a new jinja2 environment for .tex creation.
Arguments: Arguments:
@ -75,8 +75,10 @@ class TexRenderer(object):
- datadirs: list of locations of the data directory - datadirs: list of locations of the data directory
(which may contain file <datadir>/templates/<template>). (which may contain file <datadir>/templates/<template>).
- lang: main language of songbook. - lang: main language of songbook.
- encoding: if set, encoding of the template.
''' '''
self.lang = lang self.lang = lang
self.encoding = encoding
# Load templates in filesystem ... # Load templates in filesystem ...
loaders = [FileSystemLoader(os.path.join(datadir, 'templates')) loaders = [FileSystemLoader(os.path.join(datadir, 'templates'))
for datadir in datadirs] for datadir in datadirs]
@ -186,10 +188,11 @@ class TexRenderer(object):
""" """
subvariables = {} subvariables = {}
template_file = None
templatename = self.texenv.get_template(template).filename templatename = self.texenv.get_template(template).filename
try: with patacrep.encoding.open_read(
template_file = encoding.open_read(templatename, 'r') templatename,
encoding=self.encoding
) as template_file:
content = template_file.read() content = template_file.read()
subtemplates = list(find_templates(self.texenv.parse(content))) subtemplates = list(find_templates(self.texenv.parse(content)))
match = re.findall(_VARIABLE_REGEXP, content) match = re.findall(_VARIABLE_REGEXP, content)
@ -209,9 +212,6 @@ class TexRenderer(object):
jsonstring=var, jsonstring=var,
) )
) )
finally:
if template_file:
template_file.close()
return (subvariables, subtemplates) return (subvariables, subtemplates)

59
readme.md

@ -1,6 +1,4 @@
Songbook Compilation Chain # Patacrep, a songbook compilation chain
# Description
This package provides a compilation toolchain that produce LaTeX This package provides a compilation toolchain that produce LaTeX
songbook using the LaTeX songs package. A new LaTeX document class is songbook using the LaTeX songs package. A new LaTeX document class is
@ -12,45 +10,52 @@ is precised in the header.
# Python version # Python version
Patacrep is compatible with Python 3. Patacrep is only compatible with Python > 3.3.
# Download # Installation
Clone Patacrep repos: ## Using pip
As simple as
> git clone git://github.com/patacrep/patacrep.git ```
> git clone git://github.com/patacrep/patadata.git pip3 install patacrep
```
# Installation from source ## For developement
Make sure you have [pip](https://pip.pypa.io/en/latest/) installed, and then run Clone Patacrep repos:
> pip install -r Requirements.txt ```
> python3 setup.py install git clone git://github.com/patacrep/patacrep.git
cd patacrep
pip3 install -r Requirements.txt
python3 setup.py install
```
# Run ## Quick and dirty Debian (and Ubuntu?) package
> <patacrep>/songbook <songbook_file.sb> This requires [stdeb](https://github.com/astraw/stdeb) to be installed.
> <pdfreader> <songbook_file.pdf>
Look for existing songbook files in `<patadata>/books/`. For example: ```
python setup.py --command-packages=stdeb.command bdist_deb
sudo dpkg -i deb_dist/python3-patacrep_4.0.0-1_all.deb
```
> <patacrep>/songbook <patadata>/books/songbook_en.sb
> <pdfreader> songbook_en.pdf
# Quick and dirty deb packages # Run
Install `python3-stdeb`, then: ```
songbook <songbook_file.sb>
<pdfreader> <songbook_file.pdf>
```
> python3 setup.py --command-packages=stdeb.command bdist_deb Look for existing songbook files in [patadata](http://github.com/patacrep/patadata)
> sudo dpkg -i deb_dist/python3-patacrep_<version>-1_all.deb
# Documentation # More informations
- Compiled, but may be outdated: http://www.patacrep.com/data/documents/doc_en.pdf The full documentation is hosted by readthedoc, here : http://patacrep.readthedocs.org/
- Documentation repository (to update the previous one): [patacrep-doc](http://github.com/patacrep/patacrep-doc)
# Contact & Forums # Contact & Forums
* http://www.patacrep.com * http://www.patacrep.com/forum
* crep@team-on-fire.com

65
setup.py

@ -1,4 +1,4 @@
#!/usr/bin/env python #!/usr/bin/env python3
"""Installation script for songbook. """Installation script for songbook.
@ -6,57 +6,36 @@ $ python setup.py install
""" """
from patacrep import __version__ from patacrep import __version__
from setuptools import setup from setuptools import setup, find_packages
import sys setup(
import os name='patacrep',
import site version=__version__,
description='Songbook compilation chain',
author='The Songbook team',
SETUP = {"name": 'patacrep', author_email='crep@team-on-fire.com',
"version": __version__, url='https://github.com/patacrep/patacrep',
"description": 'Songbook compilation chain', packages=find_packages(),
"author": 'The Songbook team', license="GPLv2 or any later version",
"author_email": 'crep@team-on-fire.com', install_requires=[
"url": 'https://github.com/patacrep/patacrep', "unidecode", "jinja2", "chardet", "ply",
"packages": ['patacrep', 'patacrep.content', 'patacrep.latex'],
"license": "GPLv2 or any later version",
"scripts": ['songbook'],
"requires": [
"argparse", "codecs", "distutils", "fnmatch", "glob", "json",
"locale", "logging", "os", "re", "subprocess", "sys",
"textwrap", "unidecode", "jinja2", "chardet"
], ],
"install_requires": [ include_package_data=True,
"argparse", "unidecode", "jinja2", "chardet", "ply" entry_points={
'console_scripts': [
"songbook = patacrep.songbook:main",
], ],
"package_data": {'patacrep': [ 'data/latex/*', },
'data/templates/*', classifiers=[
'data/examples/*.sb',
'data/examples/*/*.sg',
'data/examples/*/*.ly',
'data/examples/*/*.jpg',
'data/examples/*/*.png',
'data/examples/*/*.png',
'data/examples/*/*/header']},
"classifiers": [
"Environment :: Console", "Environment :: Console",
"License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)", "License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)",
"Natural Language :: English", "Natural Language :: English",
"Operating System :: POSIX :: Linux", "Operating System :: POSIX :: Linux",
"Operating System :: Microsoft :: Windows", "Operating System :: Microsoft :: Windows",
"Operating System :: MacOS :: MacOS X", "Operating System :: MacOS :: MacOS X",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.3",
"Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.4",
"Topic :: Utilities", "Topic :: Utilities",
], ],
"platforms": ["GNU/Linux", "Windows", "MacOsX"] platforms=["GNU/Linux", "Windows", "MacOsX"]
} )
if sys.platform.startswith('win32'):
from shutil import copy
copy("songbook", "songbook.py")
SETUP["scripts"] = ['songbook.py']
setup(**SETUP)

156
songbook

@ -1,155 +1,9 @@
#! /usr/bin/env python3 #! /usr/bin/env python3
# -*- coding: utf-8 -*-
"""Command line tool to compile songbooks using the songbook library.""" # Do not edit this file. This file is just a helper file for development test.
# It is not part of the distributed software.
import argparse
import json
import locale
import logging
import os.path
import textwrap
import sys
from patacrep.build import SongbookBuilder, DEFAULT_STEPS
from patacrep import __version__
from patacrep import errors
from patacrep import encoding
# Logging configuration
logging.basicConfig(level=logging.INFO)
LOGGER = logging.getLogger()
# pylint: disable=too-few-public-methods
class ParseStepsAction(argparse.Action):
"""Argparse action to split a string into a list."""
def __call__(self, __parser, namespace, values, __option_string=None):
if not getattr(namespace, self.dest):
setattr(namespace, self.dest, [])
setattr(
namespace,
self.dest,
(
getattr(namespace, self.dest)
+ [value.strip() for value in values[0].split(',')]
),
)
class VerboseAction(argparse.Action):
"""Set verbosity level with option --verbose."""
def __call__(self, *_args, **_kwargs):
LOGGER.setLevel(logging.DEBUG)
def argument_parser(args):
"""Parse arguments"""
parser = argparse.ArgumentParser(description="A song book compiler")
parser.add_argument('--version', help='Show version', action='version',
version='%(prog)s ' + __version__)
parser.add_argument('book', nargs=1, help=textwrap.dedent("""\
Book to compile.
"""))
parser.add_argument('--datadir', '-d', nargs='+', type=str, action='append',
help=textwrap.dedent("""\
Data location. Expected (not necessarily required)
subdirectories are 'songs', 'img', 'latex', 'templates'.
"""))
parser.add_argument('--verbose', '-v', nargs=0, action=VerboseAction,
help=textwrap.dedent("""\
Show details about the compilation process.
"""))
parser.add_argument('--steps', '-s', nargs=1, type=str,
action=ParseStepsAction,
help=textwrap.dedent("""\
Steps to run. Default is "{steps}".
Available steps are:
"tex" produce .tex file from templates;
"pdf" compile .tex file;
"sbx" compile index files;
"clean" remove temporary files;
any string beginning with '%%' (in this case, it will be run
in a shell). Several steps (excepted the custom shell
command) can be combinend in one --steps argument, as a
comma separated string.
""".format(steps=','.join(DEFAULT_STEPS))),
default=None,
)
options = parser.parse_args(args)
return options """Command line tool to compile songbooks using the songbook library."""
def main():
"""Main function:"""
# set script locale to match user's
try:
locale.setlocale(locale.LC_ALL, '')
except locale.Error as error:
# Locale is not installed on user's system, or wrongly configured.
LOGGER.error("Locale error: {}\n".format(str(error)))
options = argument_parser(sys.argv[1:])
songbook_path = options.book[0]
basename = os.path.basename(songbook_path)[:-3]
songbook_file = None
try:
songbook_file = encoding.open_read(songbook_path)
songbook = json.load(songbook_file)
except Exception as error: # pylint: disable=broad-except
LOGGER.error(error)
LOGGER.error("Error while loading file '{}'.".format(songbook_path))
sys.exit(1)
finally:
if songbook_file:
songbook_file.close()
# Gathering datadirs
datadirs = []
if options.datadir:
# Command line options
datadirs += [item[0] for item in options.datadir]
if 'datadir' in songbook:
# .sg file
if isinstance(songbook['datadir'], str):
songbook['datadir'] = [songbook['datadir']]
datadirs += [
os.path.join(
os.path.dirname(os.path.abspath(songbook_path)),
path
)
for path in songbook['datadir']
]
# Default value
datadirs.append(os.path.dirname(os.path.abspath(songbook_path)))
songbook['datadir'] = datadirs
try:
sb_builder = SongbookBuilder(songbook, basename)
sb_builder.unsafe = True
sb_builder.build_steps(options.steps)
except errors.SongbookError as error:
LOGGER.error(error)
if LOGGER.level >= logging.INFO:
LOGGER.error(
"Running again with option '-v' may give more information."
)
sys.exit(1)
except KeyboardInterrupt:
LOGGER.warning("Aborted by user.")
sys.exit(1)
sys.exit(0)
if __name__ == '__main__': from patacrep.songbook import main
main() main()

8
stdeb.cfg

@ -1,6 +1,6 @@
[DEFAULT] [DEFAULT]
Depends: python3-jinja2, python3-pkg-resources, python3-chardet, python3-unidecode, texlive-latex-base, texlive-latex-recommended, texlive-latex-extra, lilypond, texlive-fonts-recommended Debian-Version: 1
Recommends: texlive-lang-english, texlive-lang-french, texlive-lang-portuguese, texlive-lang-spanish, texlive-fonts-extra
X-Python3-Version:
Section: tex Section: tex
Depends3: texlive, texlive-latex-base, texlive-latex-recommended, texlive-latex-extra
Recommends3: lilypond, texlive-lang-english, texlive-lang-french, texlive-lang-portuguese, texlive-lang-spanish, texlive-fonts-extra
Copyright-File: LICENSE

Loading…
Cancel
Save