format of the program

[Hepta-Col]

Hi! I am using your code to generate the program of statements. Could you please tell me:

1. what "r." means in the generated program?
2. does the dash symbol mean something special?
3. why are dates replaced by "xx"?

"program": "{ filter_str_contain_any giuseppe farina r.pole_position-string all_rows } { filter_str_contain_any italian grand prix r.race-string v0 } <END>",

Thanks!

[ousuixin]

Thank you for your attention to our work, here are my answers to your questions:

1. "r." is just used for parsing the property (we can also say it column name) from the program;
2. dash symbol is just used to split the column name and the type of column (including date, string and num);
   "r." and "-" won't be considered in the model, they just used for visualization.
3. dates replaced by "xx" means the corresponding information can not be found in the original table, for example, 1999.xx.xx means there is only year information in the table;

[Hepta-Col]

Thank you so much! Also, could you please direct me to the code where the generated programs are executed? I am curious about how you verify the correctness of the generated programs.

[ousuixin]

Hello, the execution of the program follows the generation of the program, I mean the program is generated token by token, and the generated token is processed immediately, by the function "read_token"

SASP/nsm/computer_factory.py

Lines 357 to 393 in 1fa315c

	def read_token(self, token):
	"""Read in one token, parse and execute the expression if completed."""
	if ((self.n_exp >= self.max_n_exp) or
	(self.namespace.n_var >= self.max_mem)):
	self.done = True
	new_exp = self.parse_step(token)
	# If reads in end of program, then return the last value as result.
	if self.done:
	# self.result = self.namespace.get_last_value()
	self.result = []
	for k, v in self.namespace.items():
	if v['type'] == 'bool':
	self.result.extend(v['value'])
	elif v['type'] == 'entity_list' and v['used'] == 0 and v['name'] != 'all_rows':
	self.result.append(len(v['value']) != 0)
	return self.result
	elif new_exp:
	if self.assisted:
	name = self.namespace.generate_new_name()
	result = self.eval(['define', name, new_exp])
	result['name'] = name
	self.n_exp += 1
	# If there are errors in the execution, self.eval
	# will return None. We can also give a separate negative
	# reward for errors.
	if result is None or self.exist_program(new_exp):
	self.namespace.n_var -= 1
	self.done = True
	self.result = [ERROR_TK]
	else:
	self.update_mem(new_exp, name, result)
	# result = self.eval(['define', name, ERROR_TK])
	else:
	result = self.eval(new_exp)
	return result
	else:
	return None

When an operation is not finished yet, "new_exp" is None and nothing happen. Once an operation is generated, the corresponding execution result was calculated. When the whole program generation is done, we will calculate the final execution result of the generated program, and add a bool value to "self.result".

For the program "{ filter_str_contain_any giuseppe farina r.pole_position-string all_rows } { filter_str_contain_any italian grand prix r.race-string v0 } ", the program filter the original table two times, and if the result of two filter operation is not None, we will add "True" to "self.result". And you can also take the final operation "filter_str_contain_any italian grand prix r.race-string v0" as ”hop_str_contain_any italian grand prix r.race-string v0“

[Hepta-Col]

Thanks! Now I understand how you process the programs. I have some other questions regarding programs generation though.

Here is what I am trying to do: I have another dataset of tables (all in csv format), say it has 100 (table, table description in natural language) pairs, and I want to generate 100 (table, table description in program) pairs.

According to your code, it seems that you will first preprocess table data with preprocess_example.py, with input files being files under all_csv from TabFact, intermediate file being tables.jsonl, and output file being data_shard/dev.jsonl. Then you continue preprocess table data with gen_with_func.py, with input file being tables.jsonl and output file being data_shard/test.jsonl and data_shard/train_xxx.jsonl. Next, let's say models are trained already, run test(args) in table/experiments.py to 1) create test environments; 2) decode test environments and get decode_results (programs); 3) transform decode_results and store the results to test_result.json.

So I hope to do the similar things. Since all I have is 100 (table, table description in natural language) pairs, and all I want is 100 (table, table description in program) pairs, I only need to do the followings: 1) preprocess my tables with preprocess_example.py and gen_with_func.py to generate preprocessed tables; 2) run test(args) to generate programs

I really could not go over each script to figure out all the details, so I beg your help on these questions:

1. After preprocessing csv files, I didn't see any stored information about "statements", or "table description" in "test.jsonl", but they are needed when generating operation-oriented tree and doing inference with TabBERT, so where do you store them?
2. what is example_path_untagged file? It seems that this file is needed to run preprocess_example.py. How do I prepare this file?
3. is there any quick way to achieve my goal?

Thank you very much!

[ousuixin]

Hello, Can I know the command you use with preprocess_example.py ? So that I can know what happen to you. Now I just guess it.

The usage of preprocess_example.py is like

SASP/table/tabfact/preprocess_example.py

Lines 16 to 25 in 1fa315c

	# python preprocess.py
	# --csv_dir=/home/ousuixin/Desktop/Table-Fact-Checking-master/data/all_csv
	# --used_csv_ids=/home/ousuixin/Desktop/SASP/data/tabfact/csv_ids.txt
	# --data_dict_file=/home/ousuixin/Desktop/SASP/data/tabfact/tables.jsonl
	# --min_frac_for_props=0.25
	# --corenlp_path=/home/ousuixin/Downloads/stanford-corenlp-4.2.0
	# --example_path_untagged=/home/ousuixin/Desktop/Table-Fact-Checking-master/tokenized_data/train_examples.json
	# --example_path=/home/ousuixin/Desktop/SASP/data/tabfact/examples.tagged
	# --processed_example_path=/home/ousuixin/Desktop/SASP/data/tabfact/data_shard

So you need feed a csv_ids.txt like
1-10121127-1.html.csv
1-10006830-1.html.csv
...
as input with parameter used_csv_ids. It contains the csv file names you want to use, one line per file.

What's more, you need to feed the "statement"/"table description" file in the format similar to https://github.com/wenhuchen/Table-Fact-Checking/blob/master/tokenized_data/train_examples.json
as input with parameter example_path_untagged. (The answer for the second question)

And configure the path for example_path, where the intermediate tagged file examples.tagged will be saved.

After that, the processed table would be saved according to the parameter data_dict_file and the processed "statement"/"table description" would be saved according to processed_example_path. (The answer for the first question)

For the last question, if you have enough computational resources, maybe you can try to fine-tune the pretrained model to do semantic parsing, here are some brilliant semantic parsing works with pre-trained language model like
[1] Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou: TAPEX: Table Pre-training via Learning a Neural SQL Executor. ICLR 2022
[2] Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir R. Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, Tao Yu:
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models. CoRR abs/2201.05966 (2022)
(orz) so that you don't need to do much preprocess.

[Hepta-Col]

Thank you very much! I have solved the problems above. If you don't mind, here are a few more concerns about the operators and hope that you could take a look:

1. for "max/min", it takes two arguments: sub-table and column-number, as explained in Appendix, but I notice that in the output json file, e.g., "test.json" under "runs/demo_run/", they take three arguments instead of two. Here are a few examples: { maximum v0 v3 r.crowd-number }, { maximum all_rows v1 r.caf_champion_league-number }, { maximum all_rows v1 r.gold-number }. Could you please tell me what "maximum/minimum" means when it takes three arguments?
2. similarly, for "only", it takes one argument: sub-table, as explained in Appendix, but I also notice that in the output json file it takes two arguments instead of one. Here is an example: { only all_rows v0 }. Could you please also explain this?
3. for interpreting the output "program", I notice that sometimes there are more than one root nodes after transforming the program to a tree. E.g., "{ filter_eq 1983-xx-xx r.year-date all_rows } { only all_rows v0 } { filter_str_contain_any race r.team-string v0 } <END>", "{ filter_str_contain_any viktor melantiev r.athlete-string all_rows } { filter_str_contain_any aliaksandr zhukouski r.athlete-string all_rows } { row_less v1 v0 r.rank-number } { filter_str_contain_any viktor melantiev r.athlete-string v1 } <END>". I thought there would always be a "top" operation (the final operation) that takes as argument the nearest "v?", like "v2" in the second example. How do I interpret a program like this?

Again, thank you very much for your patience and elaboration on the answers.

[ousuixin]

Hello, here are answers for questions above:

1. { maximum v0 v3 r.crowd-number } means the r.crowd-number of v3 is the max number among r.crowd-number of v0
2. { only all_rows v0 } means v0 is the only line in all_rows, actually you can ignore the first argument of the operator "only".
3. Yes, you are right, we add all intermediate execution results with bool type to "self.result", and use an operator like "and" as the top operator. Take { filter_str_contain_any viktor melantiev r.athlete-string all_rows } { filter_str_contain_any aliaksandr zhukouski r.athlete-string all_rows } { row_less v1 v0 r.rank-number } { filter_str_contain_any viktor melantiev r.athlete-string v1 } <END> as an example, the top operation is and({v2, v3!=None}).

[Hepta-Col]

Thank you! I am clear about the first and second questions. But for the third one, could you explain more about how you determine the top operation to be and(v2, v3), if all I know is the program { filter_str_contain_any viktor melantiev r.athlete-string all_rows } { filter_str_contain_any aliaksandr zhukouski r.athlete-string all_rows } { row_less v1 v0 r.rank-number } { filter_str_contain_any viktor melantiev r.athlete-string v1 } <END>? And how about "{ filter_eq 1983-xx-xx r.year-date all_rows } { only all_rows v0 } { filter_str_contain_any race r.team-string v0 } <END>"?

[ousuixin]

As I mentioned above, "self.result" is a set of bool execution result. For { filter_str_contain_any viktor melantiev r.athlete-string all_rows } { filter_str_contain_any aliaksandr zhukouski r.athlete-string all_rows } { row_less v1 v0 r.rank-number } { filter_str_contain_any viktor melantiev r.athlete-string v1 } <END>, "self.result" is {v2, v3!=None}. And for { filter_eq 1983-xx-xx r.year-date all_rows } { only all_rows v0 } { filter_str_contain_any race r.team-string v0 } <END>, "self.result" is {v1, v2!=None}.