Skip to content

Cassandrautils #4609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 113 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
3dfb76c
POC build rebased branch
nikhilbonte21 Apr 4, 2025
b44f532
POC tagOnly tag tables
nikhilbonte21 Apr 7, 2025
62297bf
POC Fix issues locally
nikhilbonte21 Apr 7, 2025
b00befd
POC disable notifications
nikhilbonte21 Apr 7, 2025
5f207ce
POC tag prop: switch mode on-demand
nikhilbonte21 Apr 8, 2025
d234c48
Added new Cassandra DAO layer for tags
suraj5077 Apr 8, 2025
cfe66a2
Merge remote-tracking branch 'origin/poctagprophashedrebased' into po…
suraj5077 Apr 8, 2025
6d30ad8
Added new Cassandra DAO layer for tags
suraj5077 Apr 8, 2025
6321a29
POC Enable notifications
nikhilbonte21 Apr 8, 2025
d0619b4
POC opt notifications
nikhilbonte21 Apr 8, 2025
1696438
Fixed AtlasClassification
suraj5077 Apr 8, 2025
8133d17
Merge remote-tracking branch 'origin/poctagprophashedrebased' into po…
suraj5077 Apr 8, 2025
acec20d
Added new get tag DB call
suraj5077 Apr 8, 2025
8d9a9d5
Clean-up
suraj5077 Apr 8, 2025
a50a70b
Added error handling to Cassandra DAO
suraj5077 Apr 8, 2025
ae23eaf
Added error handling to Cassandra DAO
suraj5077 Apr 8, 2025
19bfb9b
POC intermediate
nikhilbonte21 Apr 8, 2025
395ec97
POC multiple run options
nikhilbonte21 Apr 8, 2025
d4ad551
POC multiple options
nikhilbonte21 Apr 9, 2025
cde605e
Added async notifications
suraj5077 Apr 9, 2025
0e4212a
Merge remote-tracking branch 'origin/poctagprophashedrebased' into po…
suraj5077 Apr 9, 2025
4782e36
Changed async queue size
suraj5077 Apr 9, 2025
f004235
POC multiple options
nikhilbonte21 Apr 9, 2025
e86b9dc
Added batching for notif calls
suraj5077 Apr 9, 2025
b4c860a
Merge remote-tracking branch 'origin/poctagprophashedrebased' into po…
suraj5077 Apr 9, 2025
8084046
POC loggings
nikhilbonte21 Apr 9, 2025
14ca0ea
POC Bucket logic fix
nikhilbonte21 Apr 9, 2025
eaf063c
POC fix minimal flow
nikhilbonte21 Apr 10, 2025
eb2f9ee
POC Hande Delete propagation untested
nikhilbonte21 Apr 10, 2025
5f38a6e
POC Direct ADD tag
nikhilbonte21 Apr 12, 2025
2a9f4e9
POC support tags and reads
nikhilbonte21 Apr 14, 2025
177739a
POC Fix dao NPE
nikhilbonte21 Apr 14, 2025
e4ec63b
POC Fix auditsearch API
nikhilbonte21 Apr 14, 2025
66f57d5
POC Fix delete direct tag
nikhilbonte21 Apr 14, 2025
ad3880f
POC Fix delete direct tag
nikhilbonte21 Apr 14, 2025
b853953
POC fix asset creation
nikhilbonte21 Apr 14, 2025
7b8a321
POC Fix ES sync usual flow - except repair index
nikhilbonte21 Apr 17, 2025
c801582
POC TEMP: experimetal graph changes
nikhilbonte21 Apr 17, 2025
a1631f3
POC Fix: ES index repair
nikhilbonte21 Apr 17, 2025
cbf8b6b
POC Fix issues and atributes support
nikhilbonte21 Apr 17, 2025
57b8a5d
POC Fix read propagated issue
nikhilbonte21 Apr 17, 2025
499c5a9
Fix textMapper NPE
nikhilbonte21 Apr 17, 2025
6dec7c8
propagation text update code to talk to cassandra
abhijeet-atlan Apr 18, 2025
bd763d9
util method for propagation text update
abhijeet-atlan Apr 18, 2025
cfdaed2
MLH 289 Fix de-norm attrs when asset create with tags
nikhilbonte21 Apr 18, 2025
b798d22
MLH-276: Fix indexsearch -> classificationNames is empty
nikhilbonte21 Apr 18, 2025
82b70f3
Refactor TagDao impl
nikhilbonte21 Apr 21, 2025
972882a
MLH-279 Revert archiving and delete rows right away
nikhilbonte21 Apr 22, 2025
85c8e81
MLH-304 Optimise Select star queries to fetch only columns whichever …
nikhilbonte21 Apr 22, 2025
6d3ae7c
MLH-306 Start up error while initiating delete handler
nikhilbonte21 Apr 22, 2025
81db2f0
fixed wrong arguments for text update
abhijeet-atlan Apr 22, 2025
bdabf1d
code to handle pagination in cassandra
abhijeet-atlan Apr 22, 2025
d1cbc2a
Add skeleton code
aarshi0301 Apr 23, 2025
922a13b
code cleanup
aarshi0301 Apr 23, 2025
72da3fe
fix configs
aarshi0301 Apr 23, 2025
9107b39
fix file names
aarshi0301 Apr 23, 2025
c87efc0
Code in progress
aarshi0301 Apr 24, 2025
bf5fe30
Fixed autowiring for EntityGraphRetriever
suraj5077 Apr 24, 2025
1931673
removed redundant var
suraj5077 Apr 24, 2025
6b8f674
load atlas configs
aarshi0301 Apr 25, 2025
bf72696
flags setup
aarshi0301 Apr 25, 2025
859584d
add checks
aarshi0301 Apr 25, 2025
e74a172
do branch based deployemnt
aarshi0301 Apr 25, 2025
f771e5d
code to handle pagination in cassandra
abhijeet-atlan Apr 25, 2025
c30f1b4
fix vesrion issue
aarshi0301 Apr 25, 2025
f46e0a5
pom changes to have no conflicts in jackson library
sriram-atlan Apr 25, 2025
2b15b54
Merge remote-tracking branch 'origin/cassandrautils' into cassandrautils
sriram-atlan Apr 25, 2025
db8d4ee
allow atlasJsonProvider to bind correctly
sriram-atlan Apr 25, 2025
7323508
update aspect weaver version. use jackson's jackson-jaxrs-base lib
sriram-atlan Apr 25, 2025
399c12c
revert all jackson changes.
sriram-atlan Apr 25, 2025
f1e6645
add other properties at root
sriram-atlan Apr 25, 2025
92c4cc5
WIP: Added rollback state to EMC and added rollback for insert, update.
suraj5077 Apr 25, 2025
806b988
Complete all scenarios for Cassandra TAG rollback
suraj5077 Apr 26, 2025
7eada0c
make batch size configurable
aarshi0301 Apr 26, 2025
2fae40f
Added refresh prop task. Ensured backward compatibility.
hr2904 Apr 27, 2025
3c3aa3f
Merge branch 'poctagprophashedrebased-HR-backup' into poctagprophashe…
hr2904 Apr 27, 2025
6948864
WIP: Added ES post processing, Cassandra batch deletes, support for s…
suraj5077 Apr 27, 2025
afef36f
Fixed error and code cleanup
suraj5077 Apr 27, 2025
cadc5af
Fixed error and code cleanup
suraj5077 Apr 27, 2025
b2a8db8
Fixed error and code cleanup
suraj5077 Apr 27, 2025
90195a6
Code review comments addressed
suraj5077 Apr 28, 2025
07ad445
test
suraj5077 Apr 28, 2025
50b61bd
Fixed the getTraitNames for new optimisation logic
hr2904 Apr 28, 2025
af8f223
Added a retry mechanism in updateMetadataTimestamp due to SocketError
hr2904 Apr 28, 2025
b3431db
Merge pull request #4626 from atlanhq/MLH339Fix
hr2904 Apr 28, 2025
e241346
Ensured that cassandra keyspace and table is created. Also optimised …
hr2904 Apr 28, 2025
f1ff2e6
Merge pull request #4631 from atlanhq/MLH305
hr2904 Apr 28, 2025
f04aa0b
Merge branch 'poctagprophashedrebased' into feat/tag-rollbacks
suraj5077 Apr 28, 2025
d7dad64
code to deal with bug in cassandra and ES writes
abhijeet-atlan Apr 29, 2025
6f4a298
Added soft delete queries, refactored queries, connection pooling and…
suraj5077 Apr 29, 2025
8f99251
Merge pull request #4622 from atlanhq/feat/tag-rollbacks
suraj5077 Apr 29, 2025
dd5e26f
fixed query bindings
suraj5077 Apr 29, 2025
a7e2797
Merge pull request #4640 from atlanhq/feat/tag-rollbacks
suraj5077 Apr 29, 2025
d247afe
Fixed Cassandra RF
suraj5077 Apr 30, 2025
da292f6
Removed timeouts
suraj5077 Apr 30, 2025
7032ffd
Fixed Cassandra RF
suraj5077 Apr 30, 2025
486b900
Fixed boolean flag removePropagationsOnEntityDelete and getFeatureFla…
hr2904 Apr 30, 2025
a52606b
Merge pull request #4643 from atlanhq/poctagprophashedrebased-hr-stas…
hr2904 Apr 30, 2025
c463b2c
removed unnecessary log line
abhijeet-atlan Apr 30, 2025
c1eda6f
Merge branch 'poctagprophashedrebased' into cassandraPagination
abhijeet-atlan Apr 30, 2025
93cb2d1
Merge pull request #4623 from atlanhq/cassandraPagination
abhijeet-atlan Apr 30, 2025
c6cdf08
Revert "MLH : 340 -> Cassandra pagination bug fix"
abhijeet-atlan Apr 30, 2025
fe65056
Merge pull request #4644 from atlanhq/revert-4623-cassandraPagination
abhijeet-atlan Apr 30, 2025
999199b
code to deal with bug in cassandra and ES writes
abhijeet-atlan Apr 30, 2025
daf8a8a
Merge pull request #4645 from atlanhq/poctagprophashedrebased-ak
abhijeet-atlan Apr 30, 2025
df701b0
added paginatedTagResult class
abhijeet-atlan May 1, 2025
167be93
Merge pull request #4646 from atlanhq/poctagprophashedrebased-ak
abhijeet-atlan May 1, 2025
a58ee0c
Fixed direct update classification denorm attribute update
hr2904 May 1, 2025
3364e6c
Merge pull request #4647 from atlanhq/poctagprophash-UC-denorm-fix
hr2904 May 1, 2025
2cd72f1
Merge branch 'master' into cassandrautils
aarshi0301 May 1, 2025
08f312f
Merge branch 'poctagprophashedrebased' into cassandrautils
aarshi0301 May 1, 2025
74c53e3
Inject correct constructor
aarshi0301 May 1, 2025
6f2ce66
integrate tags
aarshi0301 May 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/maven.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ on:
- development
- master
- lineageondemand
- optimise_classifications_fetch
- cassandrautils

jobs:
build:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,6 @@ private ServicePolicies loadPolicyfromPolicyAdmin() throws RangerServiceNotFound

try {


if (serviceName.equals("atlas") && plugIn.getTypeRegistry() != null && lastUpdatedTimeInMillis == -1) {
LOG.info("PolicyRefresher(serviceName=" + serviceName + "): loading all policies for first time");
RangerRESTUtils restUtils = new RangerRESTUtils();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ public CachePolicyTransformerImpl(AtlasTypeRegistry typeRegistry) throws AtlasBa
purposeTransformer = new PurposeCachePolicyTransformer(entityRetriever);

try {
this.discoveryService = new EntityDiscoveryService(typeRegistry, graph, null, null, null, null);
this.discoveryService = new EntityDiscoveryService(typeRegistry, graph, null, null, null, null, entityRetriever);
} catch (AtlasException e) {
LOG.error("Failed to initialize discoveryService");
throw new AtlasBaseException(e.getCause());
Expand Down Expand Up @@ -599,6 +599,8 @@ private List<AtlasEntityHeader> getAtlasPolicies(String serviceName, int batchSi
List<AtlasEntityHeader> ret = new ArrayList<>();
try {
IndexSearchParams indexSearchParams = new IndexSearchParams();
indexSearchParams.setExcludeClassifications(true);

Set<String> attributes = new HashSet<>();
attributes.add(NAME);
attributes.add(ATTR_POLICY_CATEGORY);
Expand Down Expand Up @@ -673,6 +675,8 @@ private List<AtlasEntityHeader> getAtlasPolicies(String serviceName, int batchSi

private AtlasEntityHeader getServiceEntity(String serviceName) throws AtlasBaseException {
IndexSearchParams indexSearchParams = new IndexSearchParams();
indexSearchParams.setExcludeClassifications(true);

Set<String> attributes = new HashSet<>();
attributes.add(NAME);
attributes.add(ATTR_SERVICE_SERVICE_TYPE);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,20 @@ public static boolean evaluate(String key, String expectedValue) {
return ret;
}

public static String getFlag(String key) {
String ret = "";
try{
if (StringUtils.isEmpty(key))
{
return ret;
}
String value = redisService.getValue(addFeatureFlagNamespace(key));
return value;
} catch (Exception e) {
return ret;
}
}

public static void setFlag(String key, String value) {
if (StringUtils.isEmpty(key) || StringUtils.isEmpty(value))
return;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.apache.atlas.AtlasException;
import org.apache.atlas.repository.graphdb.AtlasGraph;
import org.apache.atlas.repository.graphdb.GraphDatabase;
import org.apache.atlas.repository.graphdb.janus.graphv3.AtlasJanusGraphFactory;
import org.apache.atlas.repository.graphdb.janus.serializer.BigDecimalSerializer;
import org.apache.atlas.repository.graphdb.janus.serializer.BigIntegerSerializer;
import org.apache.atlas.repository.graphdb.janus.serializer.TypeCategorySerializer;
Expand Down Expand Up @@ -149,7 +150,20 @@ static JanusGraph initJanusGraph(Configuration config) {

org.apache.commons.configuration2.Configuration conf2 = createConfiguration2(config);
try {
return JanusGraphFactory.open(conf2);
return AtlasJanusGraphFactory.open(conf2);
//return JanusGraphFactory.open(conf2);

/*JanusGraph graph = JanusGraphFactory.open(conf2);

StandardJanusGraph standardJanusGraph = (StandardJanusGraph) graph;

Field indexSerializerField = StandardJanusGraph.class.getDeclaredField("indexSerializer");
indexSerializerField.setAccessible(true);

CustomIndexSerializer customSerializer = new CustomIndexSerializer(standardJanusGraph);
indexSerializerField.set(standardJanusGraph, customSerializer);

return standardJanusGraph;*/
} catch (JanusGraphException e) {
LOG.warn("JanusGraphException: {}", e.getMessage());
if (e.getMessage().startsWith(OLDER_STORAGE_EXCEPTION)) {
Expand Down
Loading
Loading