2022-06-08 12:52:49 INFO *** initialize network ***
2022-06-08 12:52:50 INFO create new checkpoint
2022-06-08 12:52:50 INFO checkpoint: .ckpt
2022-06-08 12:52:50 INFO - [arg] dataset: dataset/mitre
2022-06-08 12:52:50 INFO - [arg] transformers_model: xlm-roberta-base
2022-06-08 12:52:50 INFO - [arg] random_seed: 1
2022-06-08 12:52:50 INFO - [arg] lr: 5e-06
2022-06-08 12:52:50 INFO - [arg] epochs: 20
2022-06-08 12:52:50 INFO - [arg] warmup_step: 0
2022-06-08 12:52:50 INFO - [arg] weight_decay: 1e-07
2022-06-08 12:52:50 INFO - [arg] batch_size: 32
2022-06-08 12:52:50 INFO - [arg] max_seq_length: 128
2022-06-08 12:52:50 INFO - [arg] fp16: False
2022-06-08 12:52:50 INFO - [arg] max_grad_norm: 1
2022-06-08 12:52:50 INFO - [arg] lower_case: False
2022-06-08 12:52:50 INFO target dataset: ['dataset/mitre']
2022-06-08 12:52:50 INFO data_name: dataset/mitre
2022-06-08 12:52:50 INFO formatting custom dataset from dataset/mitre
2022-06-08 12:52:50 INFO found following files: {'test': 'test.txt', 'train': 'train.txt', 'valid': 'valid.txt'}
2022-06-08 12:52:50 INFO note that files should be named as either `valid.txt`, `test.txt`, or `train.txt`
2022-06-08 12:52:50 INFO dataset dataset/mitre/test.txt: 747 entries
2022-06-08 12:52:50 INFO dataset dataset/mitre/train.txt: 2810 entries
2022-06-08 12:52:50 INFO dataset dataset/mitre/valid.txt: 812 entries
Some weights of the model checkpoint at xlm-roberta-base were not used when initializing XLMRobertaForTokenClassification: ['lm_head.layer_norm.weight', 'lm_head.dense.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.dense.bias']
- This IS expected if you are initializing XLMRobertaForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of XLMRobertaForTokenClassification were not initialized from the model checkpoint at xlm-roberta-base and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
Input In [2], in <cell line: 8>()
1 cfg = {'checkpoint_dir': '.ckpt',
2 'dataset': 'dataset/mitre',
3 'transformers_model': 'xlm-roberta-large',
4 'lr': 5e-6,
5 'epochs': 20,
6 'max_seq_length': 128}
7 model = cyner.TransformersNER(cfg)
----> 8 model.train()
File ~/Documents/Projects/Blog_Data_Extraction/prototype/test_CyNER/CyNER/cyner/transformers_ner.py:52, in TransformersNER.train(self)
34 cache_dir = config.get('cache_dir', None)
36 trainer = TrainTransformersNER(checkpoint_dir=checkpoint_dir,
37 dataset=dataset,
38 transformers_model=transformers_model,
(...)
49 num_worker=num_worker,
50 cache_dir=cache_dir)
---> 52 trainer.train(monitor_validation=True)
File ~/Documents/Projects/Blog_Data_Extraction/prototype/test_CyNER/CyNER/cyner/tner/model.py:292, in TrainTransformersNER.train(self, monitor_validation, batch_size_validation, max_seq_length_validation)
290 if self.args.is_trained:
291 logging.warning('finetuning model, that has been already finetuned')
--> 292 self.__setup_model_data(self.args.dataset, self.args.lower_case)
293 writer = SummaryWriter(log_dir=self.args.checkpoint_dir)
295 data_loader = {'train': self.__setup_loader('train', self.args.batch_size, self.args.max_seq_length)}
File ~/Documents/Projects/Blog_Data_Extraction/prototype/test_CyNER/CyNER/cyner/tner/model.py:155, in TrainTransformersNER.__setup_model_data(self, dataset, lower_case)
145 config = transformers.AutoConfig.from_pretrained(
146 self.args.transformers_model,
147 num_labels=len(self.label_to_id),
148 id2label=self.id_to_label,
149 label2id=self.label_to_id,
150 cache_dir=self.cache_dir)
152 self.model = transformers.AutoModelForTokenClassification.from_pretrained(
153 self.args.transformers_model, config=config)
--> 155 self.transforms = Transforms(self.args.transformers_model, cache_dir=self.cache_dir)
157 # optimizer
158 no_decay = ["bias", "LayerNorm.weight"]
File ~/Documents/Projects/Blog_Data_Extraction/prototype/test_CyNER/CyNER/cyner/tner/tokenizer.py:38, in Transforms.__init__(self, transformer_tokenizer, cache_dir)
36 def __init__(self, transformer_tokenizer: str, cache_dir: str = None):
37 """ NER specific transform pipeline """
---> 38 self.tokenizer = transformers.AutoTokenizer.from_pretrained(transformer_tokenizer, cache_dir=cache_dir)
39 self.pad_ids = {"labels": PAD_TOKEN_LABEL_ID, "input_ids": self.tokenizer.pad_token_id, "__default__": 0}
40 self.prefix = self.__sp_token_prefix()
File ~/anaconda3/envs/blogsIntel/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py:546, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
544 tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
545 if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
--> 546 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
547 else:
548 if tokenizer_class_py is not None:
File ~/anaconda3/envs/blogsIntel/lib/python3.9/site-packages/transformers/tokenization_utils_base.py:1780, in PreTrainedTokenizerBase.from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
1777 else:
1778 logger.info(f"loading file {file_path} from cache at {resolved_vocab_files[file_id]}")
-> 1780 return cls._from_pretrained(
1781 resolved_vocab_files,
1782 pretrained_model_name_or_path,
1783 init_configuration,
1784 *init_inputs,
1785 use_auth_token=use_auth_token,
1786 cache_dir=cache_dir,
1787 **kwargs,
1788 )
File ~/anaconda3/envs/blogsIntel/lib/python3.9/site-packages/transformers/tokenization_utils_base.py:1915, in PreTrainedTokenizerBase._from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, use_auth_token, cache_dir, *init_inputs, **kwargs)
1913 # Instantiate tokenizer.
1914 try:
-> 1915 tokenizer = cls(*init_inputs, **init_kwargs)
1916 except OSError:
1917 raise OSError(
1918 "Unable to load vocabulary from file. "
1919 "Please check that the provided vocabulary is accessible and not corrupted."
1920 )
File ~/anaconda3/envs/blogsIntel/lib/python3.9/site-packages/transformers/models/xlm_roberta/tokenization_xlm_roberta_fast.py:139, in XLMRobertaTokenizerFast.__init__(self, vocab_file, tokenizer_file, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, mask_token, **kwargs)
123 def __init__(
124 self,
125 vocab_file=None,
(...)
135 ):
136 # Mask token behave like a normal word, i.e. include the space before it
137 mask_token = AddedToken(mask_token, lstrip=True, rstrip=False) if isinstance(mask_token, str) else mask_token
--> 139 super().__init__(
140 vocab_file,
141 tokenizer_file=tokenizer_file,
142 bos_token=bos_token,
143 eos_token=eos_token,
144 sep_token=sep_token,
145 cls_token=cls_token,
146 unk_token=unk_token,
147 pad_token=pad_token,
148 mask_token=mask_token,
149 **kwargs,
150 )
152 self.vocab_file = vocab_file
153 self.can_save_slow_tokenizer = False if not self.vocab_file else True
File ~/anaconda3/envs/blogsIntel/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py:109, in PreTrainedTokenizerFast.__init__(self, *args, **kwargs)
106 fast_tokenizer = tokenizer_object
107 elif fast_tokenizer_file is not None and not from_slow:
108 # We have a serialization from tokenizers which let us directly build the backend
--> 109 fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
110 elif slow_tokenizer is not None:
111 # We need to convert a slow tokenizer to build the backend
112 fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
Exception: EOF while parsing a string at line 1 column 8862550
Unable to pinpoint where the problem is occurring. Could you help me with this?
Thank you.
From the
CyNER Demo.ipynbI have tried to train the model but I get this error.Output
Unable to pinpoint where the problem is occurring. Could you help me with this?
Thank you.